Objective-based control of an autonomous unmanned aerial vehicle

ABSTRACT

Techniques are described for controlling an autonomous vehicle such as an unmanned aerial vehicle (UAV) using objective-based inputs. In an embodiment, the underlying functionality of an autonomous navigation system is exposed via an application programming interface (API) allowing the UAV to be controlled through specifying a behavioral objective, for example, using a call to the API to set parameters for the behavioral objective. The autonomous navigation system can then incorporate perception inputs such as sensor data from sensors mounted to the UAV and the set parameters using a multi-objective motion planning process to generate a proposed trajectory that most closely satisfies the behavioral objective in view of certain constraints. In some embodiments, developers can utilize the API to build customized applications for the UAV. Such applications, also referred to as “skills,” can be developed, shared, and executed to control behavior of an autonomous UAV and aid in overall system improvement.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/240,394, titled “OBJECTIVE-BASED CONTROL OF AN AUTONOMOUS UNMANNEDAERIAL VEHICLE,” filed Jan. 4, 2019, and issued as U.S. Pat. No.11,048,277 on Jun. 29, 2021; which is entitled to the benefit and/orright of priority of U.S. Provisional Patent Application No. 62/621,243,titled “OBJECTIVE-BASED CONTROL OF AN AUTONOMOUS UNMANNED AERIALVEHICLE,” filed Jan. 24, 2018, the contents of each of which are herebyincorporated by reference in their entirety for all purposes. Thisapplication is therefore entitled to a priority date of Jan. 24, 2018.

TECHNICAL FIELD

The present disclosure generally relates to autonomous vehicletechnology.

BACKGROUND

Unmanned aerial vehicles (UAV) are increasingly being used as platformsfor taking images (including video) from the air. A number of UAVsystems are currently available that provide for image and video captureand remote control from a device on the ground. However, currentlyavailable systems require piloting using direct control of the UAVsimilar to other fixed wing or rotor craft. In other words, control bydirectly adjusting the pitch, roll, yaw, and power of the UAV, forexample using common control inputs such as a joystick and throttlecontrol. While effective to a degree, such control systems requireexpertise on the part of the remote pilot and are prone to crashescaused by pilot error.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows an example configuration of an autonomous vehicle in theform of an unmanned aerial vehicle (UAV) within which certain techniquesdescribed herein may be applied.

FIG. 1B shows another configuration of an autonomous vehicle in the formof a fixed-wing UAV within which certain techniques described herein maybe applied.

FIG. 2 shows a block diagram of an example navigation system that may beimplemented with the UAV of FIGS. 1A and/or 1B.

FIG. 3 shows a block diagram illustrating an example configuration forinputting objectives to the navigation system of FIG. 2 via anapplication programming interface (API).

FIG. 4 shows a diagram illustrating an example world-relative objective.

FIG. 5 shows a diagram illustrating an example vehicle-relativeobjective.

FIG. 6A-6B show diagrams illustrating an example subject-relativeobjective.

FIG. 7 shows a diagram illustrating an example subject-relativeobjective to maintain line-of-sight with a tracked subject.

FIG. 8 shows a diagram illustrating an example image-relative objective.

FIG. 9 shows a diagram illustrating an example objective to avoidbacklighting.

FIG. 10 shows a diagram illustrating an example objective to maintainscene saliency.

FIG. 11 shows a diagram illustrating an example objective to avoidcollisions with other objects.

FIG. 12 shows a block diagram illustrating an example multi-objectivemotion planning based on objective inputs received via an API.

FIG. 13 shows a block diagram illustrating certain parameters of anobjective.

FIG. 14 shows a block diagram illustrating components of example skillsin an example application built on an API.

FIG. 15 shows an example of a visual output displayed via a mobiledevice computing device.

FIG. 16 shows an example of an augmented reality (AR) visual output.

FIG. 17 shows a diagram of an example localization system with which atleast some operations described in this disclosure can be implemented.

FIG. 18 shows a diagram illustrating the concept of visual odometrybased on captured images.

FIG. 19 shows an example view of a three-dimensional (3D) occupancy mapof a physical environment.

FIG. 20 shows an example image captured by a UAV in flight through aphysical environment with associated visualizations of data regardingtracked objects based on processing of the captured image.

FIG. 21 shows a diagram illustrating an example process for estimating atrajectory of an object based on multiple images captured by a UAV.

FIG. 22 shows a diagrammatic representation of an example spatiotemporalfactor graph.

FIG. 23 shows a diagram that illustrates an example process ofgenerating an intelligent initial estimate for where a tracked objectwill appear in a subsequently captured image.

FIG. 24 shows a visualization representative of a dense per-pixelsegmentation of a captured image.

FIG. 25 shows a visualization representative of an instance segmentationof a captured image.

FIG. 26 shows a block diagram of an example UAV system including variousfunctional system components with which at least some operationsdescribed in this disclosure can be implemented.

FIG. 27 shows a block diagram of an example of a processing system inwhich at least some operations described in this disclosure can beimplemented.

DETAILED DESCRIPTION

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

Overview

To alleviate the need for direct pilot control, UAVs used as aerialimage capture platforms can be configured for autonomous operation.Achieving autonomous flight in a safe and intelligent manner involves acomplex hierarchy of physics, control systems, scene understanding, andmotion planning. The complex nature of autonomous vehicle technologycreates a high barrier of entry for application developers seeking toleverage the high capabilities of an autonomous UAV as an image captureplatform.

To address such challenges, techniques are introduced that hide theunderlying complexity of an autonomous navigations system and providefor control of an autonomous vehicle such as a UAV through specifying acollection of intuitive, high-level behavioral intentions also referredto herein “behavioral objectives” or simply as “objectives.” In someembodiments, objectives utilized to perform motion planning are exposedthrough an application programming interface (API). Developers canutilize the API to build customized applications for utilizing anautonomous UAV to capture images of the physical environment. Suchapplications may comprise or include what are referred to herein as“skills,” or “skill sets.” Skills may comprise software and/or assetsconfigured to modify objective inputs to the underlying autonomousnavigation system, thereby controlling vehicle behavior during actualflight, during simulated flight, as well as pre-flight and post-flightbehavior. For example, a developer-created skill may change and adjustthe type of data collected during a flight (image stills vs video, framerate, etc.), change and adjust objective inputs to the navigation engineduring flight, perform customized post-processing on received data afterlanding, etc.

As will be described, skills can be developed using the API, shared withother users via an online storefront, downloaded and executed by otherusers using other UAVs, tested in an online simulation environment,and/or utilized to improve operation of the autonomous control systems.

Example Implementation of an Autonomous Vehicle

FIG. 1A shows an example configuration of a UAV 100 within which certaintechniques described herein may be applied. As shown in FIG. 1A, UAV 100may be configured as a rotor-based aircraft (e.g., a “quadcopter”). Theexample UAV 100 includes propulsion and control actuators 110 (e.g.,powered rotors or aerodynamic control surfaces) for maintainingcontrolled flight, various sensors for automated navigation and flightcontrol 112, and one or more image capture devices 114 and 115 forcapturing images of the surrounding physical environment while inflight. “Images,” in this context, include both still images and capturevideo. Although not shown in FIG. 1A, UAV 100 may also include othersensors (e.g., for capturing audio) and systems for communicating withother devices (e.g., a mobile device 104) via a wireless communicationchannel 116.

In the example depicted in FIG. 1A, the image capture devices 114 and/or115 are depicted capturing an object 102 in the physical environmentthat happens to be a person. In some cases, the image capture devicesmay be configured to capture images for display to users (e.g., as anaerial video platform) and/or, as described above, may also beconfigured for capturing images for use in autonomous navigation. Inother words, the UAV 100 may autonomously (i.e., without direct humancontrol) navigate the physical environment, for example, by processingimages captured by any one or more image capture devices. While inautonomous flight, UAV 100 can also capture images using any one or moreimage capture devices that can be displayed in real time and or recordedfor later display at other devices (e.g., mobile device 104).

FIG. 1A shows an example configuration of a UAV 100 with multiple imagecapture devices configured for different purposes. In the exampleconfiguration shown in FIG. 1A, the UAV 100 includes multiple imagecapture devices 114 arranged about a perimeter of the UAV 100. The imagecapture devices 114 may be configured to capture images for use by avisual navigation system in guiding autonomous flight by the UAV 100and/or a tracking system for tracking other objects in the physicalenvironment (e.g., as described with respect to FIG. 2 ). Specifically,the example configuration of UAV 100 depicted in FIG. 1A includes anarray of multiple stereoscopic image capture devices 114 placed around aperimeter of the UAV 100 so as to provide stereoscopic image capture upto a full 360 degrees around the UAV 100.

In addition to the array of image capture devices 114, the UAV 100depicted in FIG. 1A also includes another image capture device 115configured to capture images that are to be displayed but notnecessarily used for navigation. In some embodiments, the image capturedevice 115 may be similar to the image capture devices 114 except in howcaptured images are utilized. However, in other embodiments, the imagecapture devices 115 and 114 may be configured differently to suit theirrespective roles.

In many cases, it is generally preferable to capture images that areintended to be viewed at as high a resolution as possible given certainhardware and software constraints. On the other hand, if used for visualnavigation and/or object tracking, lower resolution images may bepreferable in certain contexts to reduce processing load and providemore robust motion planning capabilities. Accordingly, in someembodiments, the image capture device 115 may be configured to capturerelatively high resolution (e.g., 3840×2160) color images while theimage capture devices 114 may be configured to capture relatively lowresolution (e.g., 320×240) grayscale images.

The UAV 100 can be configured to track one or more objects such as ahuman subject 102 through the physical environment based on imagesreceived via the image capture devices 114 and/or 115. Further the UAV100 can be configured to track image capture of such objects, forexample, for filming purposes. In some embodiments, the image capturedevice 115 is coupled to the body of the UAV 100 via an adjustablemechanism that allows for one or more degrees of freedom of motionrelative to a body of the UAV 100. The UAV 100 may be configured toautomatically adjust an orientation of the image capture device 115 soas to track image capture of an object (e.g., human subject 102) as boththe UAV 100 and object are in motion through the physical environment.In some embodiments, this adjustable mechanism may include a mechanicalgimbal mechanism that rotates an attached image capture device about oneor more axes. In some embodiments, the gimbal mechanism may beconfigured as a hybrid mechanical-digital gimbal system coupling theimage capture device 115 to the body of the UAV 100. In a hybridmechanical-digital gimbal system, orientation of the image capturedevice 115 about one or more axes may be adjusted by mechanical means,while orientation about other axes may be adjusted by digital means. Forexample, a mechanical gimbal mechanism may handle adjustments in thepitch of the image capture device 115, while adjustments in the roll andyaw are accomplished digitally by transforming (e.g., rotating, panning,etc.) the captured images so as to effectively provide at least threedegrees of freedom in the motion of the image capture device 115relative to the UAV 100.

FIG. 2 is a block diagram that illustrates an example navigation system120 that may be implemented as part of the example UAV 100 describedwith respect to FIG. 1A. The navigation system 120 may include anycombination of hardware and/or software. For example, in someembodiments, the navigation system 120 and associated subsystems, may beimplemented as instructions stored in memory and executable by one ormore processors.

As shown in FIG. 2 , the example navigation system 120 includes a motionplanning system 130 for autonomously maneuvering the UAV 100 through aphysical environment and a tracking system 140 for tracking one or moreobjects in the physical environment. The tracking subsystem 140 mayinclude one or more subsystems such as an object detection subsystem, aninstance segmentation subsystem, an identity recognition subsystem, andany other subsystems (all not shown). The purposes of such subsystemsare described in more detail later. Note that the arrangement of systemsshown in FIG. 2 is an example provided for illustrative purposes and isnot to be construed as limiting. For example, in some embodiments, thetracking system 140 may be completely separate from the navigationsystem 120. Further, the subsystems making up the navigation system 120may not be logically separated as shown in FIG. 2 .

In some embodiments, the motion planning system 130, operatingseparately or in conjunction with the tracking system 140, is configuredto generate a planned trajectory through a three-dimensional (3D) spaceof a physical environment based, for example, on images received fromimage capture devices 114 and/or 115, data from other sensors 112 (e.g.,IMU, GPS, proximity sensors, etc.), one or more control inputs 170 fromexternal sources (e.g., from a remote user, navigation application,etc.), and/or one or more specified navigation objectives. As will bedescribed in more detail, the control inputs 170 may include calls to anapplication programming interface (API) associated with navigationsystem 120. For example, API calls may be made by an application forsetting one or more navigation objectives as part of the motion planningprocess. Navigation objectives will be described in more detail later,but may include, for example, avoiding collision with other objectsand/or maneuvering to follow a particular object (e.g., an objecttracked by tracking system 140). In some embodiments, the generatedplanned trajectory is continuously or continually (i.e., at regular orirregular intervals) updated based on new perception inputs (e.g., newlycaptured images) and/or new control inputs 170 received as the UAV 100autonomously navigates the physical environment.

In some embodiments, the navigation system 120 may generate controlcommands configured to cause the UAV 100 to maneuver along the plannedtrajectory generated by the motion planning system 130. For example, thecontrol commands may be configured to control one or more controlactuators 110 (e.g., rotors and/or control surfaces) to cause the UAV100 to maneuver along the planned 3D trajectory. Alternatively, aplanned trajectory generated by the motion planning system 120 may beoutput to a separate flight controller system 160 that is configured toprocess trajectory information and generate appropriate control commandsconfigured to control the one or more control actuators 110.

The tracking system 140, operating separately or in conjunction with themotion planning system 130, may be configured to track one or moreobjects in the physical environment based, for example, on imagesreceived from image capture devices 114 and/or 115, data from othersensors 112 (e.g., IMU, GPS, proximity sensors, etc.), one or morecontrol inputs 170 from external sources (e.g., from a remote user,navigation application, etc.), and/or one or more specified trackingobjectives. Again, in some embodiments, tracking objectives may be setbased on API calls from an application, for example, based on userinputs received through the application. Tracking objects will bedescribed in more detail later, but may include, for example, adesignation by a user to track a particular detected object in thephysical environment or a standing objective to track objects of aparticular classification (e.g., people).

As alluded to above, the tracking system 140 may communicate with themotion planning system 130, for example, to maneuver the UAV 100 basedon measured, estimated, and/or predicted positions, orientations, and/ortrajectories of objects in the physical environment. For example, thetracking system 140 may communicate a navigation objective to the motionplanning system 130 to maintain a particular separation distance to atracked object that is in motion.

In some embodiments, the tracking system 140, operating separately or inconjunction with the motion planning system 130, is further configuredto generate control commands configured to cause a mechanism to adjustan orientation of any image capture devices 114/115 relative to the bodyof the UAV 100 based on the tracking of one or more objects. Such amechanism may include a mechanical gimbal or a hybrid digital-mechanicalgimbal, as previously described. For example, while tracking an objectin motion relative to the UAV 100, the tracking system 140 may generatecontrol commands configured to adjust an orientation of an image capturedevice 115 so as to keep the tracked object centered in the field ofview (FOV) of the image capture device 115 while the UAV 100 is inmotion. Similarly, the tracking system 140 may generate commands oroutput data to a digital image processor (e.g., that is part of a hybriddigital-mechanical gimbal) to transform images captured by the imagecapture device 115 to keep the tracked object centered in the FOV of theimage capture device 115 while the UAV 100 is in motion.

The UAV 100 shown in FIG. 1A and the associated navigation system 120shown in FIG. 2 are examples provided for illustrative purposes. A UAV100 in accordance with the present teachings may include more or fewercomponents than are shown. Further, the example UAV 100 depicted in FIG.1A and associated navigation system 120 depicted in FIG. 2 may includeor be part of one or more of the components of the example UAV system2600 described with respect to FIG. 26 and/or the example computerprocessing system 2700 described with respect to FIG. 27 . For example,the aforementioned navigation system 120 and associated tracking system140 may include or be part of the UAV system 2600 and/or processingsystem 2700.

While the introduced technique for objective-based control of anautonomous vehicle using an API is described in the context of an aerialvehicle such as the UAV 100 depicted in FIG. 1A, such a technique is notlimited to this context. The described technique may similarly beapplied to guide navigation and image capture by other types of vehicles(e.g., fixed-wing aircraft, automobiles, watercraft, etc.), hand-heldimage capture devices (e.g., mobile devices with integrated cameras), orto stationary image capture devices (e.g., building mounted securitycameras). For example, FIG. 1B shows an example of a fixed-wing UAV 100b. Similar to the UAV 100 described with respect to FIG. 1A, thefixed-wing UAV 100 b shown in FIG. 1B may include multiple image capturedevices 114 b arranged about a perimeter of the UAV 100 b configured tocapture images for use by a visual navigation system in guidingautonomous flight by the UAV 100 b. The example fixed-wing UAV 100 b mayalso include a subject image capture device 115 b configured to captureimages (e.g., of subject 102) that are to be displayed but notnecessarily used for navigation. For simplicity, embodiments of theintroduced technique are described herein with reference to the UAV 100of FIG. 1A; however, a person having ordinary skill in the art willrecognize that the introduced technique can be similarly applied usingthe fixed-wing UAV 100 b of FIG. 1B.

Objective-Based Control of an Autonomous Vehicle Using an API

The complex processing by a navigation system 120 to affect theautonomous behavior of a UAV 100 can be abstracted into one or morebehavioral objectives. A “behavioral objective” or “objective” in thiscontext generally refers to any sort of defined goal or targetconfigured to guide an autonomous response by the UAV 100. For example,objectives may be configured to approximate certain intentions of ahuman pilot. FIGS. 4-11 will describe some example “objectives” withinthe meaning of this term as used herein. It shall be appreciated thatthe example objectives described with respect to FIGS. 4-11 are providedfor illustrative purposes and are not to be construed as limiting. Asystem in accordance with the present discloser may be based on fewer ormore objectives than are described.

The underlying processes performed by a navigation system 120 forcausing a UAV 100 to autonomously maneuver through an environment and/orperform image capture can be exposed through an application programminginterface (API). For example, FIG. 3 shows a diagram of navigationsystem 120 including a motion planning component 130 and trackingcomponent 140, for example, as described with respect to FIG. 2 . Aspreviously discussed with respect to FIG. 2 , the navigation system 120may generate control outputs 302 such as a proposed trajectory, specificcontrol commands, and or image capture outputs based on perceptioninputs received from sensors (e.g., image capture devices 114/115 and/orother sensors 112) as well as one or more control inputs 170. In thecontext of the diagram of FIG. 3 , such control inputs may be in theform of calls to an API 300 defining parameters of one or moreobjectives 1 through N.

As will be described in more detail, the API 300 may be configured as apublic facing API that may be utilized by a developer to createapplications configured to enable certain user interactions with the UAV100 without specific knowledge of the underlying processes of thenavigation system 120 that enable autonomous behavior by the UAV 100. Insome cases, the developer creating such applications may be a“second-party” or “third-party” developer, meaning that the developermay be an entity other than the original developer of the navigationsystem 120 (or one or more internal components of the navigation system120).

World-Relative Objectives

In some embodiments, an objective may be expressed in terms relative tothe physical environment in which the UAV 100 resides. Such objectivesare referred to herein as “world-relative” objectives. An example of aworld-relative navigation objective may include maneuvering the UAV to aspecific location in the physical environment. Similarly, a“world-relative” image capture objective may include positioning the UAV100 and an associated image capture device 115 so as to capture aspecific location in the physical environment.

FIG. 4 shows a view of a map 402 of a physical environment. A specificlocation in the physical environment is indicated at marker 404. In thisexample, the location may be defined based on a global positioningcoordinate (e.g., latitude, longitude), however other types of locationindicators may similarly be applied. For example, locations in thephysical environment may similarly be defined based on a localcoordinate system (e.g., a grid coordinate for a particular city),position/orientation coordinate relative to a takeoff point of the UAV100 (i.e., a navigation coordinate), other types of location identifiers(e.g., a mailing address), a name of a point of interest (e.g., theGolden Gate Bridge) at a known location, and the like.

A target of a world-relative objective may be expressed based on any ofthe above-mentioned types of location indicators. For example, aworld-relative objective in the form of a GPS coordinate (e.g., 37.40,−122.16) may be input into the navigation system 120 of UAV 100 (e.g.,in the form of a call to API 300) to cause the UAV 100 to autonomouslymaneuver through the physical environment to the designated locationand/or direct image capture at the designated location. Note that FIG. 4shows an indirect path (as indicated by the dotted line) between acurrent position of the UAV 100 and the location designated by theworld-relative objective. Such an indirect path may be based on aproposed trajectory generated by a motion planning component 130 of thenavigation system 120 to autonomously maneuver the UAV 100 to thedesignated location 404 while satisfying other objectives such asavoiding obstacles, maintaining visual contact with a subject, etc.

World-relative objectives are described above as being defined based onlocations in the physical environment, however they may similarlyinclude other defining parameters such as relative motion (e.g., groundvelocity or air velocity), altitude (expressed as a value above mean sea(MSL), above ground level (AGL), etc.), a separation distance to certainobjects in the physical environment (e.g., lateral distance to avertical surface such as a wall), etc. For example, a particularworld-relative objective that incorporates multiple defined targets maybe semantically expressed as “fly to grid coordinate 37.40, −122.16while maintaining a velocity of 30 miles per hour and an altitude of atleast 1000 AGL.” Similarly, this objective may be expressed as threeindependent world-relative objectives. As will be described,world-relative objective(s) may be provided as inputs (e.g., in the formof calls to API 300) to the navigation system 120 of the UAV 100 tocause the UAV 100 to autonomously maneuver in a manner that attempts tomeet the objective(s) while taking into account other objectives (e.g.,avoiding collision with other objects).

Vehicle-Relative Objectives

In some embodiments, an objective may be expressed in terms relative tothe vehicle itself (e.g., UAV 100). For example, a vehicle-relativeobjective may include a target to move forward, backward, left, right,up, down, and/or rotate about one or more axes (e.g., yaw, pitch, roll,etc.) at some defined speed or acceleration (angular speed oracceleration in the case of rotation objectives). Similarly, avehicle-relative objective may include a target to adjust the positionand/or orientation of an image capture device 115 relative to the bodyof the UAV 100, for example, through the use of a gimbal mechanism.

Vehicle-relative objectives may be defined based on a vehicle-relativecoordinate system. For example, FIG. 5 depicts a representative view ofan example UAV 100 and a multi-dimensional coordinate system 502 uponwhich lateral motion (e.g., along X, Y, and Z axes) and rotationalmotion (e.g., about the X, Y, and Z axes) can be defined. Similarcoordinate system may be defined relative to the image capture device115 for defining image capture objectives.

As an illustrative example, a vehicle-relative objective may besemantically expressed as “move forward (e.g., along the Y axis) at aconstant ground speed of 3 miles per hour.” As with the world-relativeobjectives described above, vehicle-relative objective(s) may beprovided as inputs (e.g., in the form of calls to API 300) to thenavigation system 120 of the UAV 100 to cause the UAV 100 toautonomously maneuver in a manner that attempts to meet the objective(s)while taking into account other objectives (e.g., avoiding collisionwith other objects).

Subject-Relative Objectives

In some embodiments, an objective may be expressed in terms relative tosome other physical object (i.e., a subject) in the physicalenvironment. The “subject” in this context may include any type ofobject such as a person, an animal, a vehicle, a building, a landscapefeature, or any other static or dynamic physical objects present in thephysical environment. For example, a subject-relative navigationobjective may include a target to move to and/or maintain a particularposition and/or orientation relative to a tracked subject in thephysical environment. Similarly, a subject-relative image captureobjective to capture maneuver so as to capture images of the trackedsubject in the physical environment.

Subject-relative objectives may be defined, for example, inposition/orientation terms based on values for an azimuth, elevation,range, height, azimuth rate between the vehicle and the tracked subject.For example, FIGS. 6A-6B show side view and a top view (respectively)that illustrate how relative positioning between a UAV 100 and a trackedsubject (in this case a human subject 102) can be defined in terms of anelevation angle θ₁, an azimuth angle θ₂, and a range value.

Subject-relative objectives may also include targets that are definedbased on a semantic understanding of physical environment that the UAV100 and subject occupy. For example, a subject-relative objective mayinclude a target to maintain a clear line of sight between the UV 100and the tracked subject. FIG. 7 depicts an example scenario involving aUAV 100 in flight over a physical environment 720 while capturing imagesof a human subject 102. As shown in FIG. 7 , at a current time, humansubject 102 is located on an opposite side of object 730 from UAV 100;however, as indicated by dotted line 710, a view of human subject 102from an image capture device onboard UAV 100 is not occluded by object730. If the human subject 102 moves to a different position behind theobject 730, the view of the human subject 102 from the image capturedevice onboard the UAV 100 may be occluded, as indicated by dotted line712. Accordingly, to satisfy a subject-relative objective to maintainline of sight, a navigation system 120 may cause the UAV 100 to maneuver(e.g., along trajectory 706 or 704) to a different position such thatthe view of the human subject 102 is no longer occluded.

Certain techniques for tracking subjects in the physical environment aredescribed later with respect to FIGS. 20-25 ; however, in someembodiments, a motion planning system 130 may employ a specifictechnique described below in order to satisfy a subject-relativeobjective to maintain line of sight.

Consider again the scenario depicted in FIG. 7 . Based on a predictedtrajectory of human subject 102 (as indicated by arrow 716), andmeasured or estimated positions of the UAV 100 and object 730, anavigation system 120 may determine that the view of the human subject102 may become occluded by the object 730 (assuming UAV 100 remainsstationary) as indicated by the obstructed line of sight line 712. Basedon this predicted future state and a standing objective to maintain lineof sight with subject 102, the navigation system 120 may generateoutputs (e.g., a predicted trajectory and/or control commands)configured to cause the UAV 100 to maneuver to the UAV 100 to satisfythe subject-relative objective. Here, the generated output may beconfigured to cause UAV 100 to maneuver along a flight path 706 to keepthe view of human subject 102 unobstructed. Note that in this example,simply avoiding a collision with object 730 may not be sufficient tosatisfy the objective. For example, if the generated output causes theUAV 100 to maneuver along alternative flight path 704 instead of 706,its view of human subject 102 will become momentarily obstructed byobject 730, thereby failing the objective.

The process applied by the motion planning system 130 to maneuver theUAV 100 along trajectory 706 instead of 704 in order to satisfy a lineof sight objective may be based on a virtual line of sight in acomputer-generated 3D model of the physical environment. As will bedescribed the measured, estimated, and/or predicted motions of UAV 100and one or more tracked subjects may be based on localization within acomputer-generated 3D model representative of the physical environment.The navigation system 120 may then define a virtual line connectingvirtual representations of the positions of the UAV 100 and subject 102in the 3D model. Accordingly, a subject-relative objective to maintainline of sight can be interpreted with the navigation system 120 as anobjective to maneuver the UAV 100 such that the virtual line of sightline does not intersect with a virtual representation of anotherphysical object. This criterion may be specified with a certain level oftolerance (i.e., dead zone) to account for objects in motion. In otherwords, if UAV 100 and/or subject 102 are both in motion, it may beinevitable that at certain times the virtual line connecting theirrepresentations in the virtual map may intersect representations ofother objects. However, if that intersection persists for more than acertain period of time (e.g., 1 second), the navigation system 120 mayrespond by generating an output configured to cause UAV 100 to maneuverto avoid the intersection.

In FIG. 7 , the dotted line of sight 710 may represent the virtual lineof sight connecting the representations of UAV 100 and subject 102within a virtual environment (i.e., the computer-generated 3D model)representing physical environment 720. As human subject 102 begins tomove within the physical environment, the virtual line 710 connectingthe virtual representations moves as well. If the human subject 102moves behind object 730, the virtual line within the 3D map will thenintersect the corner of a virtual representation of physical object 730as indicated by dotted line 712. When this intersection occurs in thevirtual environment, the subject-relative objective to maintain visualcontact is no longer satisfied in the physical environment. Note thatthis may represent a state several seconds in the future based on apredicted motion of the UAV 100 and/or subject 102. A current orpredicted intersection of the virtual line of sight with a virtualrepresentation of a physical object will therefore cause the navigationsystem 120 to generate an output to configured to cause the UAV 100 tomaneuver to avoid the intersection. For example, the motion of thevirtual line can be tracked, and it may be determined that in order toavoid the intersection, UAV 100 should maneuver along flight path 706 asopposed to flight path 704 to keep the view of subject 102 unobstructed.

In some situations, intersection points along a virtual line can beanalyzed differently depending on their distance to the UAV 100. Thismay be based on an assumption that motion by a UAV 100 generally has agreater impact on resolving visual occlusions caused by objects that arecloser to the UAV 100. This assumption may depend on the size and/orshape of the obstructing object; however, in general, relatively minormaneuvers by UAV 100 may be sufficient to maintain line of sight with asubject around an object that is close to UAV 100. Conversely, moredrastic maneuvers by UAV 100 may be necessary to maintain line of sightaround an object that is closer to subject 102. This makes sense whenagain considering the scenario described in FIG. 7 . Although describedas a single object 730, the virtual representation of object 1030 canalso be described as multiple surfaces that intersect the virtual lineat multiple points. For example, obstructed line of sight line 712intersects a first surface of object 730 that faces UAV 100 at a firstpoint and a second surface of object 730 that faces a future position ofsubject 102 at a second point. A minor maneuver along flight path 706may be sufficient such that sight line 712 no longer intersects thefirst surface (i.e., the surface closest to UAV 100) at the first point.However, a more extended maneuver along flight path 706 may be necessarybefore sight line 712 no longer intersects the second surface (i.e., thesurface closest to subject 102) at the second point, therebyestablishing line of sight with subject 102.

In some embodiments, a subject-relative objective such maintaining lineof sight may be built into the navigation system 120 as a core objective(e.g., similar to avoiding collisions), for example, to comply with aflight regulation. For example, a UAV 100 may be subject to a regulationthat requires a human operator to maintain visual line of sight with theUAV 100. A simple control restraint on separation distance (i.e., range)between a subject (i.e., the human operator) and the UAV 100 may sufficeto an extent but will not ensure that visual line of sight ismaintained. Instead, the above described technique for maintaining lineof sight can be utilized.

Subject-relative objectives may also apply to multiple simultaneouslytracked subjects. In some cases, this may be accomplished by inputtingmultiple objectives (relative to each tracked subject) into thenavigation system and allowing the navigation system to generate aproposed trajectory to satisfy as many of the input subject-relativeobjectives as possible along with any other objectives (e.g., avoidcollisions). Alternatively, or in addition, a single objective relativeto multiple tracked subjects may be input contemplated. For example, asubject-relative objective may be defined relative to an averageposition and/or orientation of multiple tracked subjects in a scene.

Image-Relative Objectives

In some embodiments, an objective may be expressed in terms relative toimages captured by one or more image capture devices 114/115 onboard theUAV 100. For example, an image-relative objective may be defined to keepcertain tracked objects within an FOV of an image capture device114/115, keep certain tracked objects at a particular position in FOV ofthe image capture device 114/115, keep the horizon at a particularposition/orientation relative to the image capture device 114/115 etc.

FIG. 8 depicts an example image 802 captured (e.g., by an image capturedevice 114/115) of an object (e.g., a human subject 102). As suggestedin FIG. 8 , an image-relative objective may include a target, forexample, to keep the depiction of the tracked human subject 102 at aparticular coordinate in the image space of the captured image 802. Inthe example scenario depicted in FIG. 8 , a target normalized imagespace coordinate for the subject 102 may be defined as (0.5, 0.7) withcorresponding dead zones of 0.2 in the y direction and 0.7 in the xdirection.

In order to satisfy certain image-relative objectives, a computingsystem associated with UAV 100 may process images received from theimage capture devices 114/115 onboard the UAV 100 to perform an imagespace analysis of certain objects (e.g., a tracked subject or thehorizon) detected in the captured images. Specific techniques by whichimages are processed to detect objects are described with respect toFIGS. 20-25 .

Semantic-Based Objectives

In some embodiments, objectives may be based on semantic understandingof the physical environment. Examples of such objectives may includeavoiding backlighting by the sun, maintaining scene saliency (e.g.,focusing on “interesting” objects or image regions), avoiding dangerousor critical areas, tracking certain classes of objects (e.g., people vs.animals), tracking objects performing a certain activities (e.g., peoplerunning vs. standing still), landmark reasoning (e.g., avoidingobfuscation of a tracked object), overall scene understanding (e.g.,capturing an image of one object approaching another object), and thelike. It shall be appreciated that these are only a few examplesemantic-based objectives provided for illustrative purposes, and arenot to be construed as limiting. The types of semantic-based objectivesthat may be implemented may only be limited by the extent to which acomputing system associated with the UAV 100 is able to gain a semanticunderstanding of the physical environment and the multiple objectsoccupying the physical environment.

FIG. 9 shows an example scenario that illustrates a semantic-basedobjective including a target that avoids backlighting by the sun whencapturing images of a tracked object. As shown in FIG. 9 , a UAV 100 isin autonomous flight over a physical environment 920 while tracking andcapturing images of a particular object (in this case human subject102). The human subject 102 is lit by a light source 950 (in thisexample the Sun) from one side. Here, UAV 100 is shown at a currentlocation (as indicated by the solid line quadcopter) opposite the lightsource 950 relative to the human subject 102. At this current position,images captured of human subject 102 (for example within FOV 910) arelikely to be devoid of much detail of human subject 102 due to theshadow cast by the light source 950. In the case of a powerful lightsource 950 such as the Sun, the captured images may be completely washedout due to over exposure, particularly if the image capture deviceassociated with UAV 100 is oriented so as to be pointed substantially inthe direction of the light source 950.

Subjectively, backlighting during image capture is generally understoodto result in poor quality images. Accordingly, in some embodiments, asemantic-based objective may be configured to avoid backlighting. Tosatisfy such an objective, a navigation system 120 may generate anoutput (e.g., control commands or a proposed trajectory) configured tocause the UAV 100 to autonomously position itself substantially betweencertain light sources (e.g., the Sun) and a tracked subject 102 whencapturing images of the tracked subject 102. Consider again the scenariodepicted in FIG. 9 . Since UAV 100 is located opposite a major lightsource 950 while capturing images of subject 102, in order to satisfy aspecified objective, a navigation system 120 may generate control anoutput configured to cause UAV 100 to autonomously maneuver along flightpath 904 until, at a future time, UAV 100 is located substantiallybetween light source 950 and subject 102 (as indicated by the dottedline quadcopter). A method for generating such an output may include, inaddition to estimating the motions of UAV 100 and subject 102, alsoestimating a position of a light source 950. This may be accomplished ina number of ways for example, by processing images captured by an imagecapture device 114/115 associated with UAV 100 and/or based onlocalization data of known light sources (e.g., the Sun). Given globalpositioning information for UAV 100 and the current date/time, alocalization system can determine if UAV 100 is pointed towards the Sunwhile capturing images of a subject 102.

In some embodiments, a semantic-based objective may include a target tocapture images of “interesting” objects in the physical environment.This may be generally referred to as scene or visual saliency. Theattention of humans and certain other animals tends to be attracted tovisually salient stimuli. Visually salient stimuli may be based, forexample, on the closest object roughly centered in an FOV, an object inmotion, an object performing a certain activity of interest, etc.

What is deemed “interesting” or visually salient may of course differdepending on the context in which the UAV 100 is operating. As anillustrative example, an objective may be configured to cause a UAV 100track and capture images of a particular class of object (e.g., people)and/or of a particular type of activity. For example, FIG. 10 shows anexample scenario involving a UAV 100 in autonomous flight through aphysical environment 1020. In this example scenario, an objective may beconfigured to track and capture images of people that are skiing.Successfully satisfying such an objective may require detection ofobjects in the physical environment 1020 as well as a semanticunderstanding of the scene in order to distinguish a person 102 b thatis standing still or performing some other activity (e.g., walking) froma person 102 a that is skiing. Additional information regarding theprocessing of images to gain a semantic understanding of a scene isdescribed in more detail below with respect to FIGS. 20-25 .

In order to satisfy the objective, a navigation system may generate anoutput (e.g., control commands or a proposed trajectory) configured tocause the UAV 100 to follow a person skiing 102 a (when detected) andfocus image capture on that person 102 a. In some embodiments, the UAV100 may simply follow the tracked object at a set distance.Alternatively, or in addition, the UAV 100 may execute maneuvers inorder to add a dynamic quality to the captured images. For example, asshown in FIG. 10 , the UAV 100 may autonomously maneuver along a path1002 to capture the skier 102 a at different angles as the skier 102 acontinues down the slope. In some cases, such maneuvers may be based onpre-scripted flying patterns that are triggered when a particular object(e.g., a skier 102 a) is detected. Alternatively, or in addition, flightpaths that provide “interesting” shots may be learned by the system overtime by applying machine learning.

The scenario depicted in FIG. 10 is provide for illustrative purposesand is not to be construed as limiting. Another example semantic-basedobjective for visual salience may include a target tracking andcapturing images of a key individual in a team sporting event. Considerfor example, a football game involving two teams, each with multipleplayers. To capture images of the game, a semantic-based objective maybe configured to cause a UAV 100 to track and capture images of anobject of interest such as the football, a player in current possessionof the football, a player with imminent possession of the football(e.g., a receiver about to catch the football), the end zone, a referee,the coach, etc. Over the course of the game, the object or set ofobjects of interest will likely change from one moment to the next.Again, the manner in which the UAV 100 responds to satisfy the objectivemay be based on pre-scripted patterns of motion and image capture or maybe learned, for example, by analyzing professional television broadcastsof sporting events.

High-Level Behavioral Objectives

Certain objectives may be based around high-level behavior such asmaintaining a certain dynamic smoothness in proposed trajectories,avoiding exceeding dynamic airframe constrains, avoiding obstaclecollisions, prioritizing avoiding collisions with certain classes ofobjects (e.g., people), avoiding running out of storage space for imagecapture, avoiding running out of power, etc.

As an illustrative example, FIG. 11 depicts an example scenarioinvolving a UAV 100 with a high level behavioral objective to avoidcollisions with other objects. In the scenario depicted in FIG. 11 , aUAV 100 is in flight through a physical environment 1120 while capturingimages of a human subject 102. As shown in FIG. 11 , UAV 100 may be inautonomous flight along a current planned flight path 1104 to maneuverto avoid a collision with another object 1130 in the physicalenvironment while keeping human subject 102 in view (as indicated by FOVlines 1110. The example illustrated in FIG. 11 is idealized and shows arelatively large stationary object 1130 (for example a building or otherstructure), but the same concept may apply to avoid smaller mobileobjects such as a bird in flight. As shown in FIG. 11 , based on theestimated motions of UAV 100 and subject 102, a navigation system 120may generate an output (e.g., control commands or a proposed trajectory)to maneuver UAV 100 along flight path 1104 to avoid object 1130 whilekeeping human subject 102 in view (as indicated by FOV lines 1110).Notably, this scenario illustrates a combination of multiple objectives,specifically maintaining line of sight with a tracked subject (aspreviously discussed) while avoiding collision. As will be discussedfurther, the multiple objectives may be weighted differently such that anavigation system 120 favors satisfying one objective (e.g., avoidingcollision) over another (e.g., maintaining line of sight with a trackedsubject) if both cannot be satisfied concurrently.

Another example high-level behavioral objective may include autonomouslylanding the UAV 100 when a power source (e.g., batteries) powering apropulsion system (e.g., the rotors) is at or below a threshold level ofpower (e.g., charge). For example, in some embodiments, if the batterieson the UAV 100 get below a certain threshold level (e.g., 5% charge),the UAV 100 may automatically land on the ground regardless of any otheractive objectives so as to avoid a loss of control and possible crash.

Another example high-level objective may include smoothing proposedtrajectories. In many situations, particularly when performing imagecapture, abrupt changes in the direction of flight of the UAV 100 maynot be preferred. Accordingly, in some embodiments, a navigations systemmay incorporate a high-level objective to maintain a certain smoothnessin any generated proposed trajectory.

As suggested by the aforementioned examples, some of these high levelbehavioral objectives may be based around ensuring safe autonomousoperation of the UAV 100. In some cases, such objectives may be builtinto a motion planning process of a navigation system 120 so as toalways be actively considered when generating a proposed trajectory. Inother words, regardless of any objectives received through calls to theAPI 300, the motion planning system 130 of the navigations system mayalways take into account certain built-in objectives such as obstacleavoidance, dynamic airframe constraints.

Objective-Based Motion Planning Using an API

In some embodiments, a navigation system 120 (e.g., specifically amotion planning component 130) is configured to incorporate multipleobjectives at any given time to generate an output such as a proposedtrajectory that can be used to guide the autonomous behavior of the UAV100. The motion planning component 130 can take into consideration thedynamic constraints of the aircraft when generating outputs such asproposed trajectories. For example, given a similar set of objectives, aproposed trajectory for a quadcopter UAV such as UAV 100 may bedifferent than a proposed trajectory for a fixed-wing UAV such as theUAV 100 b due to the different flight capabilities of the two craft.

The trajectory generation process can include gradient-basedoptimization, gradient-free optimization, sampling, end-to-end learning,or any combination thereof. The output of this trajectory generationprocess can be a proposed trajectory over some time horizon (e.g., 10seconds) that is configured to be interpreted and utilized by a flightcontroller 160 to generate control commands that cause the UAV 100 tomaneuver according to the planned trajectory. A motion planning system130 may continually perform the trajectory generation process as newperception inputs (e.g., images or other sensor data) and objectiveinputs are received. Accordingly, the proposed trajectory may becontinually updated over some time horizon thereby enabling the UAV 100to dynamically and autonomously respond to changing conditions.

FIG. 12 shows a block diagram that illustrates an example system forobjective-based motion planning using an API. As shown in FIG. 12 , amotion planning system 130 (e.g., as discussed with respect to FIG. 2 )may generate and continually update a proposed trajectory 1220 based ontrajectory generation process involving one or more objectives (e.g., aspreviously described) and or more perception inputs 1206. The perceptioninputs 1206 may include images received from one or more image capturedevices 114/115, results of processing such images (e.g., disparityimages or depth values), and or sensor data from one or more othersensors 112 onboard the UAV 100 or associated with other computingdevices (e.g., mobile device 104) in communication with the UAV 100. Theone or more objectives 1202 utilized in the motion planning process mayinclude built-in objectives governing high-level behavior (e.g.,avoiding collision with other objects) as well as objectives based oninputs 1208.

The objective inputs 1208 may be in the form of calls to an API 300 byone or more applications 1210 associated with the UAV 100. An“application” in this context may include any set of instructions forperforming a process to control or otherwise alter the behavior of theUAV 100 through an API 300. A developer (e.g., a third-party developer)can configure an application 1210 to send a command to the UAV 100 whilein flight over a network API to alter one or more of the objectives 1202utilized by the motion planning system 130 to alter the behavior of theUAV 100. As previously noted, the UAV 100 may be configured to maintainsafe flight regardless of commands sent by an application. In otherwords, an application 1210 may not have access via the API 300 to altercertain core built-in objectives 1204 such as obstacle avoidance. TheAPI 300 can therefore be used to implement applications such as acustomize vehicle control, for example, through the use of a usercomputing device such as a mobile device 104. Such applications 1210 maybe stored in a memory associated with the UAV 100 and/or stored in amemory of another computing device (e.g., mobile device 104) that is incommunication (e.g., wireless communication) with the UAV 100.

Each of the objectives 1202 may be encoded as equations forincorporation in one or more motion planning equations utilized by themotion planning system 130 when generating a proposed trajectory tosatisfy the one or more objectives. Parameterization for the one or moreobjectives 1202 may be exposed to external entities such as externalapplications 1210 via the public facing API 300. In other words, anapplication 1210 may set values for certain objectives to affect theautonomous flight of the UAV 100 through the use of calls 1208 to theAPI 300.

Each given objective of the set of one or more objectives 1202 utilizein the motion planning process may include one or more definedparameterizations that are exposed through the API. For example, FIG. 13shows an example objective 1302 that includes a target 1304, a dead-zone1306, a weighting factor 1308, and other parameters 1310.

The target 1304 defines the goal of the particular objective that themotion planning system 130 will attempt to satisfy when proposing atrajectory 1220. For example, the target 1304 of a given objective maybe to maintain line of sight with one or more detected objects in thephysical environment as described with respect to FIG. 7 . The target1304 may similarly be associated with any of the other exampleobjectives described with respect to FIGS. 4-11 .

The dead-zone defines a region around the target 1304 in which themotion planning system 130 may not take action to correct. Thisdead-zone 1306 may be thought of as a tolerance level for satisfying agiven target 1304. For example, FIG. 8 shows an example dead-zonedefinition in the context of an image-relative objective. As shown inFIG. 8 , the target of the example image-relative objective may be tomaintain image capture of a tracked object 102 such that the trackedobject appears at a coordinate of (0.5, 0.7) in the image space of thecaptured image 802. To avoid continuous adjustments based on slightdeviations from this target, a dead-zone is defined to allow for sometolerance. For example, as shown in FIG. 8 , a dead-zone of 0.2 isdefined in the y-direction and a dead-zone of 0.7 is defined in thex-direction. In other words, as long as the tracked object 102 appearswithin an area of the image bounded by the target and respectivedead-zones, the objective is considered satisfied.

The weighting factor 1306 (also referred to as an “aggressiveness”factor) defines a relative level of impact the particular objective 1302will have on the overall trajectory generation process performed by themotion planning system 130. Recall that a particular objective 1302 maybe one of several objectives 1202 that may include competing targets. Inan ideal scenario, the motion planning system 130 will generate aproposed trajectory 1220 that perfectly satisfies all of the relevantobjectives at any given moment. For example, the motion planning system130 may generate a proposed trajectory that maneuvers the UAV 100 to aparticular GPS coordinate while following a tracked object, capturingimages of the tracked object, maintaining line of sight with the trackedobject, and avoiding collisions with other objects. In practice, such anideal scenario may be rare. Accordingly, the motion planning system 130may need to favor one objective over another when the satisfaction ofboth is impossible or impractical (for any number of reasons). Theweighting factors for each of the objectives 1202 define how they willbe considered by the motion planning system 130.

In an example embodiment, a weighting factor is numerical value on ascale of 0.0 to 1.0. A value of 0.0 for a particular objective mayindicate that the motion planning system 130 can completely ignore theobjective (if necessary), while a value of 1.0 may indicate that themotion planning system 130 will make a maximum effort to satisfy theobjective while maintaining safe flight. A value of 0.0 may similarly beassociated with an inactive objective and may be set to zero, forexample, in response to toggling by an application 1210 of the objectivefrom an active state to an inactive state. Low weighting factor values(e.g., 0.0-0.4) may be set for certain objectives that are based aroundsubjective or aesthetic targets such as maintaining visual saliency inthe captured images. Conversely, higher weighting factor values (e.g.,0.5-1.0) may be set for more critical objectives such as avoiding acollision with another object.

In some embodiments, the weighting factor values 1308 may remain staticas a proposed trajectory is continually updated while the UAV 100 is inflight. Alternatively, or in addition, weighting factors for certainobjectives may dynamically change based on changing conditions, whilethe UAV 100 is in flight. For example, an objective to avoid an areaassociated with depth value calculations in captured images (e.g., dueto low light conditions) may have a variable weighting factor thatincreases or decreases based on other perceived threats to the safeoperation of the UAV 100. In some embodiments, an objective may beassociated with multiple weighting factor values that change dependingon how the objective is to be applied. For example, a collisionavoidance objective may utilize a different weighting factor dependingon the class of a detected object that is to be avoided. As anillustrative example, the system may be configured to more heavily favoravoiding a collision with a person or animal as opposed to avoiding acollision with a building or tree.

In some embodiments, a notification is returned to an API caller (e.g.,an application 1210) in the event that an objective is requested (e.g.,via a call 1208 to an API 300), but not satisfied (e.g., due tocompeting objectives, vehicle constraints, or other reasons). The API300 may provide an endpoint for providing this notification so that thecaller (e.g., an application 1210) can take appropriate action such asnotifying a user, adjusting the requested objective, etc.

Skills Built on the API

As previously discussed, applications (e.g., applications 1210) can bebuilt on a public facing API 300 to augment the behavior of a UAV 100and/or an experience of a user interacting with the UAV 100. In someembodiments, particularly in the context of a UAV 100 with image capturecapabilities, applications can be developed around sets of instructionsand assets that enable high-level autonomous behavior by the UAV 100.These instructions and/or assets may govern various aspects of thebehavior of the UAV 100, the capture and processing of images by the UAV100, and user interactions with the UAV 100. As previously mentioned,these sets of instructions and/or assets are referred to as “skills.”

As shown in FIG. 14 , an application 1410 may include one or more skills1 through M. Further, each skill may include instructions related tocertain navigation and/or image capture objectives 1420-1 through M,imaging effects 1422-1, visualizations 1424-1 through M, and otherfeatures 1426-1 through M.

For example, with respect to skill 1, objective(s) 1420-1 may includeinstructions for modifying the objectives utilized by a motion planningsystem 130 of the UAV 100, for example, by generating calls to an API300 to set and/or modify certain parameters of one or more objectives.These instructions may govern the motion of the UAV 100 as well as otherbehavioral aspects such as object tracking, adjusting the orientation ofan image capture device 115, etc.

Imaging effects 1422-1 may include instructions and/or assets forprocessing images captured by an image capture device 114/115 to changethe appearance of captured images. Imaging effects may include anymanipulations made to the captured images such scaling, geometrictransformations (2D and/or 3D), transparency operations, splicing and/orcropping, sharpening, color correction, contrast adjustment, filters,etc. For example, a developer may configure a skill to present optionsto a user to select various pre-defined imaging effects to apply in realtime as the UAV is in flight and capturing images and/or as part of apost-production process. Alternatively, or in addition, a developer mayconfigure a skill to automatically apply certain imaging effects (inreal time or post-production) based on contextual cues in the capturedimages. For example, a skill may be configured to apply a particularimaging effect (e.g., a predefined filter) to captured images inresponse to detecting a tracking a particular class of object oractivity in the captured images.

Visualizations 1422-1 may include instructions and/or assets forproviding visual output to a user. For example, visualizations 1422-1may include augmented reality (AR) object descriptions that can berendered in real-time or near-real-time (e.g., within milliseconds) togenerate AR overlays that are displayed relative to tracked objects orother elements in the physical environment. An example of an augmentedreality interface that may be implemented in conjunction with a UAV 100is described with respect to FIG. 16 .

In some embodiments, the UAV 100 may include onboard memory for storingone or more skills as well as a sandboxed execution environmentexecuting the skills. For example, the sandboxed execution environmentmay be configured such that executing skills impact the behavior of theUAV 100 through calls to the API 300, but otherwise do not impactoperation of the core navigation system 130. In this way, active skillscan be safely changed while the UAV 100 is in flight without negativelyimpacting the safe flight of the UAV 100. In some embodiments, skillsmay execute at an external device such as a mobile device 104 and/or atan external data processing service such as a cloud-based computingenvironment utilizing multiple machines. In such cases, execution of theskills may generate outputs (e.g., control commands) that are thentransmitted to the UAV 100 (e.g., via a wireless communication link) tocontrol certain behavior of the UAV 100.

Skills can be configured to handle certain inputs from external sourcesto govern any of the aforementioned behaviors of a UAV 100. For example,a skill can be configured to receive inputs from a mobile device 104(e.g., based on inputs by a user), from another UAV, from acloud-computing services, or from any other external sources. Suchinputs may cause the skill to govern behavior by the UAV 100 such asmaneuvers or additional objectives that reside in a null-space of a setof objectives specified by the skill and/or alterations of a set ofactive objectives associated with a skill, in their set-point,dead-zone, or weighting factor settings.

In some embodiments, skills can be configured to include adjustablesettings that can be set (e.g., based on input from a user via anapplication) while the UAV 100 is in flight. For example, a skill basedaround tracking and capturing images of objects in the physicalenvironment can be configured to respond to a user input identifying aparticular object to track. Such a user input may be received via aninterface similar to the AR interface described with respect to FIG. 16.

Simulation Environment for Developing Skills

In some embodiments, users (e.g., developers and/or end-users) may beallowed to develop their own skills, for example, by using anapplication at a mobile device 104. Without requiring a deepunderstanding of the complex processes involved in the autonomousbehavior of the UAV 100, users may develop customized skills thatcombine various behavioral objectives with imaging effects,visualizations, etc., as previously discussed. These components may bepresented via an application as tools that can be selected andconfigured by a user to create customized skills.

A simulation environment may be implemented to aid users in thedevelopment of customized skills. The simulation environment may be anonline environment that offers a realistic physics based representationof the UAV 100 along with its various relevant systems (e.g., imagecapture devices, sensors, navigation system, etc.). The simulationenvironment can be configured to accurately simulate a behavioralresponse, by a UAV 100, to customized skills. Using the simulationenvironment, users can create and/or modify any code or asset in a skillto create customized skill. Users can then execute their customizedskills using the simulation environment and view a simulated response ofthe UAV 100 to the skill in the context of a physical environment withother objects present. For example, a user may wish to simulate aresponse by a UAV 100 to a customized skill based around tracking andcapturing images of objects in motion. The simulation environment maysimulate such objects in motion that are then detected by the simulatedperception systems (e.g., image capture devices, etc.) onboard thesimulated UAV 100. Users can edit, save, and/or upload the skills thatthey create, for example, to for use in a real UAV 100 and/or to sharewith other users.

Application Store for Sharing Skills

In some embodiments, skills created by users can be uploaded to anonline storefront (e.g., an “app store”) for sharing or sale. Otherusers can then browse listings of skills that others have created anddownload selected skills for simulation in a simulation environmentand/or use with a real UAV 100. Users can comment on skills whilesharing them through the online storefront. Further, when uploadingskills to the online storefront, users can also upload images (e.g.,video) captured when executing the skills using a device such as a UAV100. In this way, other users can observe what to expect when using anuploaded skill.

In some embodiments, the public online storefront may be regulated tomaintain certain standards around the skills created by users. Forexample, the public online storefront may include an automated frameworkthat tests uploaded skills (e.g., using a simulation environment) toensure that the uploaded skills at least do not interfere with the safeoperation of a UAV 100. The automated framework may also screen uploadedskills for other criteria such as compliance with applicableregulations, privacy concerns, etc.

Skills as a Learning and Verification Tool for Improving AutonomousBehavior

The results of simulation and implementation of skills created bymultiple users (developers and/or end-users) may serve as a valuabledataset. The data may be used to improve the autonomous behavior of aUAV 100 and by extension user experiences, for example, throughconducting studies and/or training machine learning processes. In someembodiments, a simulation environment (similar to as previouslydescribed) can be implemented as a verification tools to run largenumbers of simulations of skills created by other users. Data collectedfrom the running of these simulations can be used, for example, to studyand verify vehicle software integrity, perception changes, and/orimprovements for future software updates.

An example machine learning application includes learning when usersselect certain skills in response to contextual factors such asinformation in captured images, position of tracked objects, semanticcues in the surrounding physical environment, or any other perceivedfeedback. Information learned through observing user selections ofskills can be used to guide the automatic selection of skills while aUAV 100 is in flight based on any of the aforementioned factors. Anotherexample machine learning application includes creating a skill thatcauses a specific type of motion or response and using data collectedabout the conditions in which users activate such skills to informlearned policies and/or modify certain parameters associated with theskill. For example, a skill configured to cause a UAV 100 to “squeezethrough a gap” can be used to learn, based on perception inputs, whichobjects in the physical world are safe to fly near. Such a specificskill may also be used to determine whether built-in obstacle avoidancebehavior is configured too aggressively or too conservatively.

Visual Outputs Based on Skills

In some embodiments, skills can be configured to cause display of avisual output to user, for example, based on images captured from a UAV100. FIG. 15 shows an example of a visual output 1502 displayed via amobile device 104 in the form of a tablet display device. As indicatedin FIG. 15 , the mobile device 104 may be communicatively coupled with aUAV 100 in flight through a physical environment 1500 via a wirelesscommunication link 116. The UAV 100 autonomously navigates the physicalenvironment based on one or more navigation objectives, for example,associated with an active skill, as previously discussed. The skill mayfurther include instructions and/or assets configured to cause displayof a visual output 1502 via the mobile device 104. The visual output1502 may include a live video feed from an image capture device 114/115onboard the UAV 100, recorded video from an image capture device 114/115onboard the UAV 100, a rendering of a computer-generated model of thephysical environment 1500 (e.g., based on data from the image capturedevice 114/115 and/or other sensors 112 onboard the UAV 100), and thelike. As previously discussed, in some embodiments, a skill may includeinstructions and/or assets for processing captured images to applyimaging effects and/or other visualizations. For example, display output1502 depicts a composite of a live video feed of the physicalenvironment 1500 from the UAV 100 with added graphical elements (e.g.,imaging effects, graphical overlays, interactive graphical interfacefeatures, etc.).

In some embodiments, a visual output based on a skill can includegenerated and displayed “augmentations.” Devices configured foraugmented reality (AR devices) can deliver to a user a direct orindirect view of a physical environment which includes objects that areaugmented (or supplemented) by computer-generated sensory outputs suchas sound, video, graphics, or any other data that may augment (orsupplement) a user's perception of the physical environment. Forexample, data gathered or generated by a tracking system 140 regarding atracked object in the physical environment can be displayed to a user inthe form of graphical overlays via an AR device. Such augmentations maybe displayed via the AR device while the UAV 100 is in flight throughthe physical environment and actively tracking the object and/or as anaugmentation to video recorded by the UAV 100 after the flight hascompleted. Examples of AR devices that may be utilized to implement suchfunctionality include smartphones, tablet computers, laptops, headmounted display devices (e.g., Microsoft HoloLens™, Google Glass™),virtual retinal display devices, heads up display (HUD) devices invehicles, etc. For example, the previously mentioned mobile device 104may be configured as an AR device. Note that for illustrative simplicitythe term “AR device” is used herein to describe any type of devicecapable of presenting augmentations (visible, audible, tactile, etc.) toa user. The term “AR device” shall be understood to also include devicesnot commonly referred to as AR devices such as virtual reality (VR)headset devices (e.g., Oculus Rift™).

FIG. 16 shows an example view 1600 of a physical environment 1610 aspresented at a display of an AR device. For example, the view 1600 maycorrespond with display 1502 presented via a mobile tablet device 104 asshown in FIG. 15 . The view 1600 of the physical environment 1610 shownin FIG. 16 may be generated based on images captured by one or moreimage capture devices 114/115 of a UAV 100 and be displayed to a uservia the AR device in real-time or near-real-time as the UAV 100 isflying through the physical environment 1610 capturing the images. Asshown in FIG. 16 , one or more augmentations may be presented to theuser in the form of augmenting graphical overlays 1620 a, 1622 a, 1624a, 1626 a, and 1620 b associated with objects (e.g., bikers 1640 a and1640 b) in the physical environment 1610. For example, in an embodiment,the aforementioned augmenting graphical overlays may be generated andcomposited with video captured by UAV 100 as the UAV 100 tracks biker1640 a. The composite including the captured video and the augmentinggraphical overlays may be displayed to the user via a display of the ARdevice (e.g., a smartphone). In other embodiments, the AR device mayinclude a transparent display (e.g., a head mounted display) throughwhich the user can view the surrounding physical environment 1610. Thetransparent display may comprise a waveguide element made of alight-transmissive material through which projected images of one ormore of the aforementioned augmenting graphical overlays are propagatedand directed at the eyes of the user such that the projected imagesappear to the user to overlay the user's view of the physicalenvironment 1610 and correspond with particular objects or points in thephysical environment.

In some embodiments augmentations may include labels with informationassociated with objects detected in the physical environment 1610. Forexample, FIG. 16 illustrates a scenario in which UAV 100 has detectedand is tracking a first biker 1640 a and a second biker 1640 b. Inresponse, one or more augmenting graphical overlays associated with thetracked objects may be displayed via the AR device at pointscorresponding to the locations of the bikers 1640 a-b as they appear inthe captured image.

In some embodiments, augmentations may indicate specific objectinstances that are tracked by UAV 100. In the illustrative exampleprovided in FIG. 16 , such augmentations are presented as augmentinggraphical overlays 1620 a-b in the form of boxes that surround thespecific object instances 1640 a-b (respectively). This is just anexample provided for illustrative purposes. Indications of objectinstances may be presented using other types of augmentations (visual orotherwise).

In some embodiments, augmentations may include identifying informationassociated with detected objects. For example, augmenting graphicaloverlays 1622 a-b include names of the tracked bikers 1640 a-b(respectively). Further, augmenting graphical overlay 1622 a includes apicture of biker 1640 a. In some embodiments, information such as thepicture of the biker 1640 a may be automatically pulled from an externalsource such as a social media platform (e.g., Facebook™, Twitter™,Instagram™, etc.). Although not shown in FIG. 16 , augmentations mayalso include avatars associated with identified people. Avatars mayinclude 3D graphical reconstructions of the tracked person (e.g., basedon captured images and other sensor data), generative “bitmoji” frominstance segmentations, or any other type of generated graphicsrepresentative of tracked objects.

In some embodiments, augmentation may include information regarding anactivity or state of the tracked object. For example, augmentinggraphical overlay 1622 a includes information regarding the speed,distance traveled, and current heading of biker 1640 a. Otherinformation regarding the activity of a tracked object may similarly bedisplayed.

In some embodiments, augmentations may include visual effects that trackor interact with tracked objects. For example, FIG. 16 shows anaugmenting graphical overlay 1624 a in the form of a projection of a 3Dtrajectory (e.g., current, past, and/or future) associated with biker1640 a. In some embodiments, trajectories of multiple tracked objectsmay be presented as augmentations. Although not shown in FIG. 16 ,augmentations may also include other visual effects such as halos,fireballs, dropped shadows, ghosting, multi-frame snapshots, etc.

Semantic knowledge of objects in the physical environment may alsoenable new AR user interaction paradigms. In other words, certainaugmentations may be interactive and allow a user to control certainaspects of the flight of the UAV 100 and/or image capture by the UAV100. Illustrative examples of interactive augmentations may include aninteractive follow button that appears above moving objects. Forexample, in the scenario depicted in FIG. 16 , a UAV is tracking themotion of both bikers 1640 a and 1640 b, but is actively following(i.e., at a substantially constant separation distance) the first biker1640 a. This is indicated in the augmenting graphical overlay 1622 athat states “currently following.” Note that a corresponding overlay1622 b associated with the second biker 1640 b includes an interactiveelement (e.g., a “push to follow” button), that when pressed by a user,would cause the UAV 100 to stop following biker 1640 a and beginfollowing biker 1640 b. Similarly, overlay 1622 a includes aninteractive element (e.g., a “cancel” button), that when pressed by auser, would cause the UAV 100 to stop following biker 1640 a. In such asituation, the UAV 100 may revert to some default autonomous navigationobjective, for example, following the path the bikers are traveling onbut not any one biker in particular.

Other similar interactive augmentations may also be implemented. Forexample, although not shown in FIG. 16 , users may inspect certainobjects, for example, by interacting with the visual depictions of theobjects as presented by the AR device. For example, if the AR deviceincludes a touch screen display, a user may cause the UAV 100 to followthe object simply by touching a region of the screen corresponding tothe displayed object. This may also be applied to static objects thatare not in motion. For example, by interacting with a region of thescreen of an AR device corresponding to the displayed path 1650, an ARinterface may display information regarding the path (e.g., source,destination, length, material, map overlay, etc.) or may cause the UAVto travel along the path at a particular altitude.

The size and geometry of detected objects may be taken intoconsideration when presenting augmentations. For example, in someembodiments, an interactive control element may be displayed as a ringabout a detected object in an AR display. For example, FIG. 16 shows acontrol element 1626 a shown as a ring that appears to encircle thefirst biker 1640. The control element 1626 a may respond to userinteractions to control an angle at which UAV 100 captures images of thebiker 1640 a. For example, in a touch screen display context, a user mayswipe their finger over the control element 1626 a to cause the UAV 100to revolve about the biker 1640 a (e.g., at a substantially constantrange) even as the biker 1640 a is in motion. Other similar interactiveelements may be implemented to allow the user to zoom image captured inor out, pan from side to side, etc.

Localization

A navigation system 120 of a UAV 100 may employ any number of othersystems and techniques for localization. FIG. 17 shows an illustrationof an example localization system 1000 that may be utilized to guideautonomous navigation of a vehicle such as UAV 100. In some embodiments,the positions and/or orientations of the UAV 100 and various otherphysical objects in the physical environment can be estimated using anyone or more of the subsystems illustrated in FIG. 17 . By trackingchanges in the positions and/or orientations over time (continuously orat regular or irregular time intervals (i.e., continually)), the motions(e.g., velocity, acceleration, etc.) of UAV 100 and other objects mayalso be estimated. Accordingly, any systems described herein fordetermining position and/or orientation may similarly be employed forestimating motion.

As shown in FIG. 17 , the example localization system 1700 may includethe UAV 100, a global positioning system (GPS) comprising multiple GPSsatellites 1702, a cellular system comprising multiple cellular antennae1704 (with access to sources of localization data 1706), a Wi-Fi systemcomprising multiple Wi-Fi access points 1708 (with access to sources oflocalization data 1706), and/or a mobile device 104 operated by a user106.

Satellite-based positioning systems such as GPS can provide effectiveglobal position estimates (within a few meters) of any device equippedwith a receiver. For example, as shown in FIG. 17 , signals received ata UAV 100 from satellites of a GPS system 1702 can be utilized toestimate a global position of the UAV 100. Similarly, positions relativeto other devices (e.g., a mobile device 104) can be determined bycommunicating (e.g., over a wireless communication link 116) andcomparing the global positions of the other devices.

Localization techniques can also be applied in the context of variouscommunications systems that are configured to transmit communicationssignals wirelessly. For example, various localization techniques can beapplied to estimate a position of UAV 100 based on signals transmittedbetween the UAV 100 and any of cellular antennae 1704 of a cellularsystem or Wi-Fi access points 1708, 1710 of a Wi-Fi system. Knownpositioning techniques that can be implemented include, for example,time of arrival (ToA), time difference of arrival (TDoA), round triptime (RTT), angle of Arrival (AoA), and received signal strength (RSS).Moreover, hybrid positioning systems implementing multiple techniquessuch as TDoA and AoA, ToA and RSS, or TDoA and RSS can be used toimprove the accuracy.

Some Wi-Fi standards, such as 802.11ac, allow for RF signal beamforming(i.e., directional signal transmission using phased-shifted antennaarrays) from transmitting Wi-Fi routers. Beamforming may be accomplishedthrough the transmission of RF signals at different phases fromspatially distributed antennas (a “phased antenna array”) such thatconstructive interference may occur at certain angles while destructiveinterference may occur at others, thereby resulting in a targeteddirectional RF signal field. Such a targeted field is illustratedconceptually in FIG. 17 by dotted lines 1712 emanating from Wi-Firouters 1710.

An inertial measurement unit (IMU) may be used to estimate positionand/or orientation of device. An IMU is a device that measures avehicle's angular velocity and linear acceleration. These measurementscan be fused with other sources of information (e.g., those discussedabove) to accurately infer velocity, orientation, and sensorcalibrations. As described herein, a UAV 100 may include one or moreIMUs. Using a method commonly referred to as “dead reckoning,” an IMU(or associated systems) may estimate a current position based onpreviously measured positions using measured accelerations and the timeelapsed from the previously measured positions. While effective to anextent, the accuracy achieved through dead reckoning based onmeasurements from an IMU quickly degrades due to the cumulative effectof errors in each predicted current position. Errors are furthercompounded by the fact that each predicted position is based on acalculated integral of the measured velocity. To counter such effects,an embodiment utilizing localization using an IMU may includelocalization data from other sources (e.g., the GPS, Wi-Fi, and cellularsystems described above) to continually update the last known positionand/or orientation of the object. Further, a nonlinear estimationalgorithm (one embodiment being an “extended Kalman filter”) may beapplied to a series of measured positions and/or orientations to producea real-time prediction of the current position and/or orientation basedon assumed uncertainties in the observed data. Kalman filters arecommonly applied in the area of aircraft navigation, guidance, andcontrols.

Computer vision may be used to estimate the position and/or orientationof a capturing camera (and by extension a device to which the camera iscoupled) as well as other objects in the physical environment. The term,“computer vision” in this context may generally refer to any method ofacquiring, processing, analyzing and “understanding” captured images.Computer vision may be used to estimate position and/or orientationusing a number of different methods. For example, in some embodiments,raw image data received from one or more image capture devices (onboardor remote from the UAV 100) may be received and processed to correct forcertain variables (e.g., differences in camera orientation and/orintrinsic parameters (e.g., lens variations)). As previously discussedwith respect to FIG. 1 , the UAV 100 may include two or more imagecapture devices 114/115. By comparing the captured image from two ormore vantage points (e.g., at different time steps from an image capturedevice in motion), a system employing computer vision may calculateestimates for the position and/or orientation of a vehicle on which theimage capture device is mounted (e.g., UAV 100) and/or of capturedobjects in the physical environment (e.g., a tree, building, etc.).

Computer vision can be applied to estimate position and/or orientationusing a process referred to as “visual odometry.” FIG. 18 illustratesthe working concept behind visual odometry at a high level. A pluralityof images are captured in sequence as an image capture device movesthrough space. Due to the movement of the image capture device, theimages captured of the surrounding physical environment change fromframe to frame. In FIG. 18 , this is illustrated by initial imagecapture FOV 1852 and a subsequent image capture FOV 1854 captured as theimage capture device has moved from a first position to a secondposition over a period of time. In both images, the image capture devicemay capture real world physical objects, for example, the house 1880and/or the person 1802. Computer vision techniques are applied to thesequence of images to detect and match features of physical objectscaptured in the FOV of the image capture device. For example, a systememploying computer vision may search for correspondences in the pixelsof digital images that have overlapping FOV. The correspondences may beidentified using a number of different methods such as correlation-basedand feature-based methods. As shown in, in FIG. 18 , features such asthe head of a human subject 1802 or the corner of the chimney on thehouse 1880 can be identified, matched, and thereby tracked. Byincorporating sensor data from an IMU (or accelerometer(s) orgyroscope(s)) associated with the image capture device to the trackedfeatures of the image capture, estimations may be made for the positionand/or orientation of the image capture relative to the objects 1880,1802 captured in the images. Further, these estimates can be used tocalibrate various other systems, for example, through estimatingdifferences in camera orientation and/or intrinsic parameters (e.g.,lens variations) or IMU biases and/or orientation. Visual odometry maybe applied at both the UAV 100 and any other computing device such as amobile device 104 to estimate the position and/or orientation of the UAV100 and/or other objects. Further, by communicating the estimatesbetween the systems (e.g., via a wireless communication link 116)estimates may be calculated for the respective positions and/ororientations relative to each other. Position and/or orientationestimates based in part on sensor data from an on board IMU mayintroduce error propagation issues. As previously stated, optimizationtechniques may be applied to such estimates to counter uncertainties. Insome embodiments, a nonlinear estimation algorithm (one embodiment beingan “extended Kalman filter”) may be applied to a series of measuredpositions and/or orientations to produce a real-time optimizedprediction of the current position and/or orientation based on assumeduncertainties in the observed data. Such estimation algorithms can besimilarly applied to produce smooth motion estimations.

In some embodiments, data received from sensors onboard UAV 100 can beprocessed to generate a 3D map of the surrounding physical environmentwhile estimating the relative positions and/or orientations of the UAV100 and/or other objects within the physical environment. This processis sometimes referred to as simultaneous localization and mapping(SLAM). In such embodiments, using computer vision processing, a systemin accordance with the present teaching can search for densecorrespondence between images with overlapping FOV (e.g., images takenduring sequential time steps and/or stereoscopic images taken at thesame time step). The system can then use the dense correspondences toestimate a depth or distance to each pixel represented in each image.These depth estimates can then be used to continually update a generated3D model of the physical environment taking into account motionestimates for the image capture device (i.e., UAV 100) through thephysical environment.

In some embodiments, a 3D model of the surrounding physical environmentmay be generated as a 3D occupancy map that includes multiple voxelswith each voxel corresponding to a 3D volume of space in the physicalenvironment that is at least partially occupied by a physical object.For example, FIG. 19 shows an example view of a 3D occupancy map 1902 ofa physical environment including multiple cubical voxels. Each of thevoxels in the 3D occupancy map 1902 correspond to a space in thephysical environment that is at least partially occupied by a physicalobject. A navigation system 120 of a UAV 100 can be configured tonavigate the physical environment by planning a 3D trajectory 1920through the 3D occupancy map 1902 that avoids the voxels. In someembodiments, this 3D trajectory 1920 planned using the 3D occupancy map1902 can be updated by applying an image space motion planning process.In such an embodiment, the planned 3D trajectory 1920 of the UAV 100 isprojected into an image space of captured images for analysis relativeto certain identified high cost regions (e.g., regions having invaliddepth estimates).

Computer vision may also be applied using sensing technologies otherthan cameras, such as light detection and ranging (LIDAR) technology.For example, a UAV 100 equipped with LIDAR may emit one or more laserbeams in a scan up to 360 degrees around the UAV 100. Light received bythe UAV 100 as the laser beams reflect off physical objects in thesurrounding physical world may be analyzed to construct a real time 3Dcomputer model of the surrounding physical world. Depth sensing throughthe use of LIDAR may in some embodiments augment depth sensing throughpixel correspondence as described earlier. Further, images captured bycameras (e.g., as described earlier) may be combined with the laserconstructed 3D models to form textured 3D models that may be furtheranalyzed in real time or near real time for physical object recognition(e.g., by using computer vision algorithms).

The computer vision-aided localization techniques described above maycalculate the position and/or orientation of objects in the physicalworld in addition to the position and/or orientation of the UAV 100. Theestimated positions and/or orientations of these objects may then be fedinto a motion planning system 130 of the navigation system 120 to planpaths that avoid obstacles while satisfying certain objectives (e.g., aspreviously described). In addition, in some embodiments, a navigationsystem 120 may incorporate data from proximity sensors (e.g.,electromagnetic, acoustic, and/or optics based) to estimate obstaclepositions with more accuracy. Further refinement may be possible withthe use of stereoscopic computer vision with multiple cameras, asdescribed earlier.

The localization system 1000 of FIG. 17 (including all of the associatedsubsystems as previously described) is only one example of a systemconfigured to estimate positions and/or orientations of a UAV 100 andother objects in the physical environment. A localization system 1700may include more or fewer components than shown, may combine two or morecomponents, or may have a different configuration or arrangement of thecomponents. Some of the various components shown in FIG. 17 may beimplemented in hardware, software or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits.

Object Tracking

A UAV 100 can be configured to track one or more objects, for example,to enable intelligent autonomous flight. The term “objects” in thiscontext can include any type of physical object occurring in thephysical world. Objects can include dynamic objects such as a people,animals, and other vehicles. Objects can also include static objectssuch as landscape features, buildings, and furniture. Further, certaindescriptions herein may refer to a “subject” (e.g., human subject 102).The terms “subject” as used in this disclosure may simply refer to anobject being tracked using any of the disclosed techniques. The terms“object” and “subject” may therefore be used interchangeably.

With reference to FIG. 2 , A tracking system 140 associated with a UAV100 can be configured to track one or more physical objects based onimages of the objects captured by image capture devices (e.g., imagecapture devices 114 and/or 115) onboard the UAV 100. While a trackingsystem 140 can be configured to operate based only on input from imagecapture devices, the tracking system 140 can also be configured toincorporate other types of information to aid in the tracking. Forexample, various other techniques for measuring, estimating, and/orpredicting the relative positions and/or orientations of the UAV 100and/or other objects are described with respect to FIGS. 17-29 .

In some embodiments, a tracking system 140 can be configured to fuseinformation pertaining to two primary categories: semantics and 3Dgeometry. As images are received, the tracking system 140 may extractsemantic information regarding certain objects captured in the imagesbased on an analysis of the pixels in the images. Semantic informationregarding a captured object can include information such as an object'scategory (i.e., class), location, shape, size, scale, pixelsegmentation, orientation, inter-class appearance, activity, and pose.In an example embodiment, the tracking system 140 may identify generallocations and categories of objects based on captured images and thendetermine or infer additional more detailed information about individualinstances of objects based on further processing. Such a process may beperformed as a sequence of discrete operations, a series of paralleloperations, or as a single operation. For example, FIG. 20 shows anexample image 2020 captured by a UAV in flight through a physicalenvironment. As shown in FIG. 20 , the example image 2020 includescaptures of two physical objects, specifically, two people present inthe physical environment. The example image 2020 may represent a singleframe in a series of frames of video captured by the UAV. A trackingsystem 140 may first identify general locations of the captured objectsin the image 2020. For example, pixel map 2030 shows two dotscorresponding to the general locations of the captured objects in theimage. These general locations may be represented as image coordinates.The tracking system 140 may further process the captured image 2020 todetermine information about the individual instances of the capturedobjects. For example, pixel map 2040 shows a result of additionalprocessing of image 2020 identifying pixels corresponding to theindividual object instances (i.e., people in this case). Semantic cuescan be used to locate and identify objects in captured images as well asassociate identified objects occurring in multiple images. For example,as previously mentioned, the captured image 2020 depicted in FIG. 20 mayrepresent a single frame in a sequence of frames of a captured video.Using semantic cues, a tracking system 140 may associate regions ofpixels captured in multiple images as corresponding to the same physicalobject occurring in the physical environment.

In some embodiments, a tracking system 140 can be configured to utilize3D geometry of identified objects to associate semantic informationregarding the objects based on images captured from multiple views inthe physical environment. Images captured from multiple views mayinclude images captured by multiple image capture devices havingdifferent positions and/or orientations at a single time instant. Forexample, each of the image capture devices 114 shown mounted to a UAV100 in FIG. 1A may include cameras at slightly offset positions (toachieve stereoscopic capture). Further, even if not individuallyconfigured for stereoscopic image capture, the multiple image capturedevices 114 may be arranged at different positions relative to the UAV100, for example, as shown in FIG. 1A. Images captured from multipleviews may also include images captured by an image captured device atmultiple time instants as the image capture device moves through thephysical environment. For example, any of the image capture devices 114and/or 115 mounted to UAV 100 will individually capture images frommultiple views as the UAV 100 moves through the physical environment.

Using an online visual-inertial state estimation system, a trackingsystem 140 can determine or estimate a trajectory of the UAV 100 as itmoves through the physical environment. Thus, the tracking system 140can associate semantic information in captured images, such as locationsof detected objects, with information about the 3D trajectory of theobjects, using the known or estimated 3D trajectory of the UAV 100. Forexample, FIG. 21 shows a trajectory 2110 of a UAV 100 moving through aphysical environment. As the UAV 100 moves along trajectory 2110, theone or more image capture devices (e.g., devices 114 and/or 115) captureimages of the physical environment at multiple views 2112 a-c. Includedin the images at multiple views 2112 a-c are captures of an object suchas a human subject 102. By processing the captured images at multipleviews 2112 a-c, a trajectory 2120 of the object can also be resolved.

Object detections in captured images create rays from a center positionof a capturing camera to the object along which the object lies, withsome uncertainty. The tracking system 140 can compute depth measurementsfor these detections, creating a plane parallel to a focal plane of acamera along which the object lies, with some uncertainty. These depthmeasurements can be computed by a stereo vision algorithm operating onpixels corresponding with the object between two or more camera imagesat different views. The depth computation can look specifically atpixels that are labeled to be part of an object of interest (e.g., asubject 102). The combination of these rays and planes over time can befused into an accurate prediction of the 3D position and velocitytrajectory of the object over time.

While a tracking system 140 can be configured to rely exclusively onvisual data from image capture devices onboard a UAV 100, data fromother sensors (e.g., sensors on the object, on the UAV 100, or in theenvironment) can be incorporated into this framework when available.Additional sensors may include GPS, IMU, barometer, magnetometer, andcameras at other devices such as a mobile device 104. For example, a GPSsignal from a mobile device 104 held by a person can provide roughposition measurements of the person that are fused with the visualinformation from image capture devices onboard the UAV 100. An IMUsensor at the UAV 100 and/or a mobile device 104 can provideacceleration and angular velocity information, a barometer can providerelative altitude, and a magnetometer can provide heading information.Images captured by cameras at a mobile device 104 held by a person canbe fused with images from cameras onboard the UAV 100 to estimaterelative pose between the UAV 100 and the person by identifying commonfeatures captured in the images. Various other techniques for measuring,estimating, and/or predicting the relative positions and/or orientationsof the UAV 100 and/or other objects are described with respect to FIGS.17-25 .

In some embodiments, data from various sensors are input into aspatiotemporal factor graph to probabilistically minimize totalmeasurement error. FIG. 22 shows a diagrammatic representation of anexample spatiotemporal factor graph 2200 that can be used to estimate a3D trajectory of an object (e.g., including pose and velocity overtime). In the example spatiotemporal factor graph 2200 depicted in FIG.22 , variable values such as the pose and velocity (represented as nodes(2202 and 2204 respectively)) connected by one or more motion modelprocesses (represented as nodes 2206 along connecting edges). Forexample, an estimate or prediction for the pose of the UAV 100 and/orother object at time step 1 (i.e., variable X(1)) may be calculated byinputting estimated pose and velocity at a prior time step (i.e.,variables X(0) and V(0)) as well as various perception inputs such asstereo depth measurements and camera image measurements via one or moremotion models. A spatiotemporal factor model can be combined with anoutlier rejection mechanism wherein measurements deviating too far froman estimated distribution are thrown out. In order to estimate a 3Dtrajectory from measurements at multiple time instants, one or moremotion models (or process models) are used to connect the estimatedvariables between each time step in the factor graph. Such motion modelscan include any one of constant velocity, zero velocity, decayingvelocity, and decaying acceleration. Applied motion models may be basedon a classification of a type of object being tracked and/or learnedusing machine learning techniques. For example, a cyclist is likely tomake wide turns at speed, but is not expected to move sideways.Conversely, a small animal such as a dog may exhibit a moreunpredictable motion pattern.

In some embodiments, a tracking system 140 can generate an intelligentinitial estimate for where a tracked object will appear in asubsequently captured image based on a predicted 3D trajectory of theobject. FIG. 23 shows a diagram that illustrates this concept. As shownin FIG. 23 , a UAV 100 is moving along a trajectory 2310 while capturingimages of the surrounding physical environment, including of a humansubject 102. As the UAV 100 moves along the trajectory 2310, multipleimages (e.g., frames of video) are captured from one or more mountedimage capture devices 114/115. FIG. 23 shows a first FOV of an imagecapture device at a first pose 2340 and a second FOV of the imagecapture device at a second pose 2342. In this example, the first pose2340 may represent a previous pose of the image capture device at a timeinstant t(0) while the second pose 2342 may represent a current pose ofthe image capture device at a time instant t(1). At time instant t(0),the image capture device captures an image of the human subject 102 at afirst 3D position 2360 in the physical environment. This first position2360 may be the last known position of the human subject 102. Given thefirst pose 2340 of the image capture device, the human subject 102 whileat the first 3D position 2360 appears at a first image position 2350 inthe captured image. An initial estimate for a second (or current) imageposition 2352 can therefore be made based on projecting a last known 3Dtrajectory 2320 a of the human subject 102 forward in time using one ormore motion models associated with the object. For example, predictedtrajectory 2320 b shown in FIG. 23 represents this projection of the 3Dtrajectory 2320 a forward in time. A second 3D position 2362 (at timet(1)) of the human subject 102 along this predicted trajectory 2320 bcan then be calculated based on an amount of time elapsed from t(0) tot(1). This second 3D position 2362 can then be projected into the imageplane of the image capture device at the second pose 2342 to estimatethe second image position 2352 that will correspond to the human subject102. Generating such an initial estimate for the position of a trackedobject in a newly captured image narrows down the search space fortracking and enables a more robust tracking system, particularly in thecase of a UAV 100 and/or tracked object that exhibits rapid changes inposition and/or orientation.

In some embodiments, the tracking system 140 can take advantage of twoor more types of image capture devices onboard the UAV 100. For example,as previously described with respect to FIG. 1A, the UAV 100 may includeimage capture device 114 configured for visual navigation as well as animage captured device 115 for capturing images that are to be viewed.The image capture devices 114 may be configured for low-latency,low-resolution, and high FOV, while the image capture device 115 may beconfigured for high resolution. An array of image capture devices 114about a perimeter of the UAV 100 can provide low-latency informationabout objects up to 360 degrees around the UAV 100 and can be used tocompute depth using stereo vision algorithms. Conversely, the otherimage capture device 115 can provide more detailed images (e.g., highresolution, color, etc.) in a limited FOV.

Combining information from both types of image capture devices 114 and115 can be beneficial for object tracking purposes in a number of ways.First, the high-resolution color information from an image capturedevice 115 can be fused with depth information from the image capturedevices 114 to create a 3D representation of a tracked object. Second,the low-latency of the image capture devices 114 can enable moreaccurate detection of objects and estimation of object trajectories.Such estimates can be further improved and/or corrected based on imagesreceived from a high-latency, high resolution image capture device 115.The image data from the image capture devices 114 can either be fusedwith the image data from the image capture device 115, or can be usedpurely as an initial estimate.

By using the image capture devices 114, a tracking system 140 canachieve tracking of objects up to 360 degrees around the UAV 100. Thetracking system 140 can fuse measurements from any of the image capturedevices 114 or 115 when estimating a relative position and/ororientation of a tracked object as the positions and orientations of theimage capture devices 114 and 115 change over time. The tracking system140 can also orient the image capture device 115 to get more accuratetracking of specific objects of interest, fluidly incorporatinginformation from both image capture modalities. Using knowledge of whereall objects in the scene are, the UAV 100 can exhibit more intelligentautonomous flight.

As previously discussed, the high-resolution image capture device 115may be mounted to an adjustable mechanism such as a gimbal that allowsfor one or more degrees of freedom of motion relative to the body of theUAV 100. Such a configuration is useful in stabilizing image capture aswell as tracking objects of particular interest. An active gimbalmechanism configured to adjust an orientation of a higher-resolutionimage capture device 115 relative to the UAV 100 so as to track aposition of an object in the physical environment may allow for visualtracking at greater distances than may be possible through use of thelower-resolution image capture devices 114 alone. Implementation of anactive gimbal mechanism may involve estimating the orientation of one ormore components of the gimbal mechanism at any given time. Suchestimations may be based on any of hardware sensors coupled to thegimbal mechanism (e.g., accelerometers, rotary encoders, etc.), visualinformation from the image capture devices 114/115, or a fusion based onany combination thereof.

A tracking system 140 may include an object detection system fordetecting and tracking various objects. Given one or more classes ofobjects (e.g., humans, buildings, cars, animals, etc.), the objectdetection system may identify instances of the various classes ofobjects occurring in captured images of the physical environment.Outputs by the object detection system can be parameterized in a fewdifferent ways. In some embodiments, the object detection systemprocesses received images and outputs a dense per-pixel segmentation,where each pixel is associated with a value corresponding to either anobject class label (e.g., human, building, car, animal, etc.) and/or alikelihood of belonging to that object class. For example, FIG. 24 showsa visualization 2404 of a dense per-pixel segmentation of a capturedimage 2402 where pixels corresponding to detected objects 2410 a-bclassified as humans are set apart from all other pixels in the image2402. Another parameterization may include resolving the image locationof a detected object to a particular image coordinate (e.g., as shown atmap 2030 in FIG. 20 ), for example, based on centroid of therepresentation of the object in a received image.

In some embodiments, the object detection system can utilize a deepconvolutional neural network for object detection. For example, theinput may be a digital image (e.g., image 2402), and the output may be atensor with the same spatial dimension. Each slice of the output tensormay represent a dense segmentation prediction, where each pixel's valueis proportional to the likelihood of that pixel belonging to the classof object corresponding to the slice. For example, the visualization2404 shown in FIG. 24 may represent a particular slice of theaforementioned tensor where each pixel's value is proportional to thelikelihood that the pixel corresponds with a human. In addition, thesame deep convolutional neural network can also predicts the centroidlocations for each detected instance, as described in the followingsection.

A tracking system 140 may also include an instance segmentation systemfor distinguishing between individual instances of objects detected bythe object detection system. In some embodiments, the process ofdistinguishing individual instances of detected objects may includeprocessing digital images captured by the UAV 100 to identify pixelsbelonging to one of a plurality of instances of a class of physicalobjects present in the physical environment and captured in the digitalimages. As previously described with respect to FIG. 24 , a denseper-pixel segmentation algorithm can classify certain pixels in an imageas corresponding to one or more classes of objects. This segmentationprocess output may allow a tracking system 140 to distinguish theobjects represented in an image and the rest of the image (i.e., abackground). For example, the visualization 2404 distinguishes pixelsthat correspond to humans (e.g., included in region 2412) from pixelsthat do not correspond to humans (e.g., included in region 2430).However, this segmentation process does not necessarily distinguishbetween individual instances of the detected objects. A human viewingthe visualization 2404 may conclude that the pixels corresponding tohumans in the detected image actually correspond to two separate humans;however, without further analysis, a tracking system may 140 be unableto make this distinction.

Effective object tracking may involve distinguishing pixels thatcorrespond to distinct instances of detected objects. This process isknown as “instance segmentation.” FIG. 25 shows an example visualization2504 of an instance segmentation output based on a captured image 2502.Similar to the dense per-pixel segmentation process described withrespect to FIG. 24 , the output represented by visualization 2504distinguishes pixels (e.g., included in regions 2512 a-c) thatcorrespond to detected objects 2510 a-c of a particular class of objects(in this case humans) from pixels that do not correspond to such objects(e.g., included in region 2530). Notably, the instance segmentationprocess goes a step further to distinguish pixels corresponding toindividual instances of the detected objects from each other. Forexample, pixels in region 2512 a correspond to a detected instance of ahuman 2510 a, pixels in region 2512 b correspond to a detected instanceof a human 2510 b, and pixels in region 2512 c correspond to a detectedinstance of a human 2510 c.

Distinguishing between instances of detected objects may be based on ananalysis of pixels corresponding to detected objects. For example, agrouping method may be applied by the tracking system 140 to associatepixels corresponding to a particular class of object to a particularinstance of that class by selecting pixels that are substantiallysimilar to certain other pixels corresponding to that instance, pixelsthat are spatially clustered, pixel clusters that fit anappearance-based model for the object class, etc. Again, this processmay involve applying a deep convolutional neural network to distinguishindividual instances of detected objects.

Instance segmentation may associate pixels corresponding to particularinstances of objects; however, such associations may not be temporallyconsistent. Consider again, the example described with respect to FIG.25 . As illustrated in FIG. 25 , a tracking system 140 has identifiedthree instances of a certain class of objects (i.e., humans) by applyingan instance segmentation process to a captured image 2502 of thephysical environment. This example captured image 2502 may representonly one frame in a sequence of frames of captured video. When a secondframe is received, the tracking system 140 may not be able to recognizenewly identified object instances as corresponding to the same threepeople 2510 a-c as captured in image 2502.

To address this issue, the tracking system 140 can include an identityrecognition system. An identity recognition system may process receivedinputs (e.g., captured images) to learn the appearances of instances ofcertain objects (e.g., of particular people). Specifically, the identityrecognition system may apply a machine-learning appearance-based modelto digital images captured by one or more image capture devices 114/115associated with a UAV 100. Instance segmentations identified based onprocessing of captured images can then be compared against suchappearance-based models to resolve unique identities for one or more ofthe detected objects.

Identity recognition can be useful for various different tasks relatedto object tracking. As previously alluded to, recognizing the uniqueidentities of detected objects allows for temporal consistency. Further,identity recognition can enable the tracking of multiple differentobjects (as will be described in more detail). Identity recognition mayalso facilitate object persistence that enables re-acquisition ofpreviously tracked objects that fell out of view due to limited FOV ofthe image capture devices, motion of the object, and/or occlusion byanother object. Identity recognition can also be applied to performcertain identity-specific behaviors or actions, such as recording videowhen a particular person is in view.

In some embodiments, an identity recognition process may employ a deepconvolutional neural network to learn one or more effectiveappearance-based models for certain objects. In some embodiments, theneural network can be trained to learn a distance metric that returns alow distance value for image crops belonging to the same instance of anobject (e.g., a person), and a high distance value otherwise.

In some embodiments, an identity recognition process may also includelearning appearances of individual instances of objects such as people.When tracking humans, a tracking system 140 may be configured toassociate identities of the humans, either through user-input data orexternal data sources such as images associated with individualsavailable on social media. Such data can be combined with detailedfacial recognition processes based on images received from any of theone or more image capture devices 114/115 onboard the UAV 100. In someembodiments, an identity recognition process may focus on one or morekey individuals. For example, a tracking system 140 associated with aUAV 100 may specifically focus on learning the identity of a designatedowner of the UAV 100 and retain and/or improve its knowledge betweenflights for tracking, navigation, and/or other purposes such as accesscontrol.

In some embodiments, a tracking system 140 may be configured to focustracking on a specific object detected in captured images. In such asingle-object tracking approach, an identified object (e.g., a person)is designated for tracking while all other objects (e.g., other people,trees, buildings, landscape features, etc.) are treated as distractorsand ignored. While useful in some contexts, a single-object trackingapproach may have some disadvantages. For example, an overlap intrajectory, from the point of view of an image capture device, of atracked object and a distractor object may lead to an inadvertent switchin the object being tracked such that the tracking system 140 beginstracking the distractor instead. Similarly, spatially close falsepositives by an object detector can also lead to inadvertent switches intracking.

A multi-object tracking approach addresses these shortcomings, andintroduces a few additional benefits. In some embodiments, a uniquetrack is associated with each object detected in the images captured bythe one or more image capture devices 114/115. In some cases, it may notbe practical, from a computing standpoint, to associate a unique trackwith every single object that is captured in the images. For example, agiven image may include hundreds of objects, including minor featuressuch as rocks or leaves of trees. Instead, unique tracks may beassociate with certain classes of objects that may be of interest from atracing standpoint. For example, the tracking system 140 may beconfigured to associate a unique track with every object detected thatbelongs to a class that is generally mobile (e.g., people, animals,vehicles, etc.).

Each unique track may include an estimate for the spatial location andmovement of the object being tracked (e.g., using the spatiotemporalfactor graph described earlier) as well as its appearance (e.g., usingthe identity recognition feature). Instead of pooling together all otherdistractors (i.e., as may be performed in a single object trackingapproach), the tracking system 140 can learn to distinguish between themultiple individual tracked objects. By doing so, the tracking system140 may render inadvertent identity switches less likely. Similarly,false positives by the object detector can be more robustly rejected asthey will tend to not be consistent with any of the unique tracks.

An aspect to consider when performing multi-object tracking includes theassociation problem. In other words, given a set of object detectionsbased on captured images (including parameterization by 3D location andregions in the image corresponding to segmentation), an issue arisesregarding how to associate each of the set of object detections withcorresponding tracks. To address the association problem, the trackingsystem 140 can be configured to associate one of a plurality of detectedobjects with one of a plurality of estimated object tracks based on arelationship between a detected object and an estimate object track.Specifically, this process may involve computing a “cost” value for oneor more pairs of object detections and estimate object tracks. Thecomputed cost values can take into account, for example, the spatialdistance between a current location (e.g., in 3D space and/or imagespace) of a given object detection and a current estimate of a giventrack (e.g., in 3D space and/or in image space), an uncertainty of thecurrent estimate of the given track, a difference between a givendetected object's appearance and a given track's appearance estimate,and/or any other factors that may tend to suggest an association betweena given detected object and given track. In some embodiments, multiplecost values are computed based on various different factors and fusedinto a single scalar value that can then be treated as a measure of howwell a given detected object matches a given track. The aforementionedcost formulation can then be used to determine an optimal associationbetween a detected object and a corresponding track by treating the costformulation as an instance of a minimum cost perfect bipartite matchingproblem, which can be solved using, for example, the Hungarianalgorithm.

In some embodiments, effective object tracking by a tracking system 140may be improved by incorporating information regarding a state of anobject. For example, a detected object such as a human may be associatedwith any one or more defined states. A state in this context may includean activity by the object such as sitting, standing, walking, running,or jumping. In some embodiments, one or more perception inputs (e.g.,visual inputs from image capture devices 114/115) may be used toestimate one or more parameters associated with detected objects. Theestimated parameters may include an activity type, motion capabilities,trajectory heading, contextual location (e.g., indoors vs. outdoors),interaction with other detected objects (e.g., two people walkingtogether, a dog on a leash held by a person, a trailer pulled by a car,etc.), and any other semantic attributes.

Generally, object state estimation may be applied to estimate one ormore parameters associated with a state of a detected object based onperception inputs (e.g., images of the detected object captured by oneor more image capture devices 114/115 onboard a UAV 100 or sensor datafrom any other sensors onboard the UAV 100). The estimated parametersmay then be applied to assist in predicting the motion of the detectedobject and thereby assist in tracking the detected object. For example,future trajectory estimates may differ for a detected human depending onwhether the detected human is walking, running, jumping, riding abicycle, riding in a car, etc. In some embodiments, deep convolutionalneural networks may be applied to generate the parameter estimates basedon multiple data sources (e.g., the perception inputs) to assist ingenerating future trajectory estimates and thereby assist in tracking.

As previously alluded to, a tracking system 140 may be configured toestimate (i.e., predict) a future trajectory of a detected object basedon past trajectory measurements and/or estimates, current perceptioninputs, motion models, and any other information (e.g., object stateestimates). Predicting a future trajectory of a detected object isparticularly useful for autonomous navigation by the UAV 100. Effectiveautonomous navigation by the UAV 100 may depend on anticipation offuture conditions just as much as current conditions in the physicalenvironment. Through a motion planning process, a navigation system ofthe UAV 100 may generate control commands configured to cause the UAV100 to maneuver, for example, to avoid a collision, maintain separationwith a tracked object in motion, and/or satisfy any other navigationobjectives.

Predicting a future trajectory of a detected object is generally arelatively difficult problem to solve. The problem can be simplified forobjects that are in motion according to a known and predictable motionmodel. For example, an object in free fall is expected to continue alonga previous trajectory while accelerating at rate based on a knowngravitational constant and other known factors (e.g., wind resistance).In such cases, the problem of generating a prediction of a futuretrajectory can be simplified to merely propagating past and currentmotion according to a known or predictable motion model associated withthe object. Objects may of course deviate from a predicted trajectorygenerated based on such assumptions for a number of reasons (e.g., dueto collision with another object). However, the predicted trajectoriesmay still be useful for motion planning and/or tracking purposes.

Dynamic objects such as people and animals, present a more difficultchallenge when predicting future trajectories because the motion of suchobjects is generally based on the environment and their own free will.To address such challenges, a tracking system 140 may be configured totake accurate measurements of the current position and motion of anobject and use differentiated velocities and/or accelerations to predicta trajectory a short time (e.g., seconds) into the future andcontinually update such prediction as new measurements are taken.Further, the tracking system 140 may also use semantic informationgathered from an analysis of captured images as cues to aid ingenerating predicted trajectories. For example, a tracking system 140may determine that a detected object is a person on a bicycle travelingalong a road. With this semantic information, the tracking system 140may form an assumption that the tracked object is likely to continuealong a trajectory that roughly coincides with a path of the road. Asanother related example, the tracking system 140 may determine that theperson has begun turning the handlebars of the bicycle to the left. Withthis semantic information, the tracking system 140 may form anassumption that the tracked object will likely turn to the left beforereceiving any positional measurements that expose this motion. Anotherexample, particularly relevant to autonomous objects such as people oranimals is to assume that that the object will tend to avoid collisionswith other objects. For example, the tracking system 140 may determine atracked object is a person heading on a trajectory that will lead to acollision with another object such as a light pole. With this semanticinformation, the tracking system 140 may form an assumption that thetracked object is likely to alter its current trajectory at some pointbefore the collision occurs. A person having ordinary skill willrecognize that these are only examples of how semantic information maybe utilized as a cue to guide prediction of future trajectories forcertain objects.

In addition to performing an object detection process in one or morecaptured images per time frame, the tracking system 140 may also beconfigured to perform a frame-to-frame tracking process, for example, todetect motion of a particular set or region of pixels in images atsubsequent time frames (e.g., video frames). Such a process may involveapplying a mean-shift algorithm, a correlation filter, and/or a deepnetwork. In some embodiments, frame-to-frame tracking may be applied bya system that is separate from an object detection system whereinresults from the frame-to-frame tracking are fused into a spatiotemporalfactor graph. Alternatively, or in addition, an object detection systemmay perform frame-to-frame tracking if, for example, the system hassufficient available computing resources (e.g., memory). For example, anobject detection system may apply frame-to-frame tracking throughrecurrence in a deep network and/or by passing in multiple images at atime. A frame-to-frame tracking process and object detection process canalso be configured to complement each other, with one resetting theother when a failure occurs.

As previously discussed, the tracking system 140 may be configured toprocess images (e.g., the raw pixel data) received from one or moreimage capture devices 114/115 onboard a UAV 100. Alternatively, or inaddition, the tracking system 140 may also be configured to operate byprocessing disparity images. A “disparity image” may generally beunderstood as an image representative of a disparity between two or morecorresponding images. For example, a stereo pair of images (e.g., leftimage and right image) captured by a stereoscopic image capture devicewill exhibit an inherent offset due to the slight difference in positionof the two or more cameras associated with the stereoscopic imagecapture device. Despite the offset, at least some of the objectsappearing in one image should also appear in the other image; however,the image locations of pixels corresponding to such objects will differ.By matching pixels in one image with corresponding pixels in the otherand calculating the distance between these corresponding pixels, adisparity image can be generated with pixel values that are based on thedistance calculations. Such a disparity image will tend to highlightregions of an image that correspond to objects in the physicalenvironment since the pixels corresponding to the object will havesimilar disparities due to the object's 3D location in space.Accordingly, a disparity image, that may have been generated byprocessing two or more images according to a separate stereo algorithm,may provide useful cues to guide the tracking system 140 in detectingobjects in the physical environment. In many situations, particularlywhere harsh lighting is present, a disparity image may actually providestronger cues about the location of objects than an image captured fromthe image capture devices 114/115. As mentioned, disparity images may becomputed with a separate stereo algorithm. Alternatively, or inaddition, disparity images may be output as part of the same deepnetwork applied by the tracking system 140. Disparity images may be usedfor object detection separately from the images received from the imagecapture devices 114/115, or they may be combined into a single networkfor joint inference.

In general, a tracking system 140 (e.g., including an object detectionsystem and/or an associated instance segmentation system) may be primaryconcerned with determining which pixels in a given image correspond toeach object instance. However, these systems may not consider portionsof a given object that are not actually captured in a given image. Forexample, pixels that would otherwise correspond with an occluded portionof an object (e.g., a person partially occluded by a tree) may not belabeled as corresponding to the object. This can be disadvantageous forobject detection, instance segmentation, and/or identity recognitionbecause the size and shape of the object may appear in the capturedimage to be distorted due to the occlusion. To address this issue, thetracking system 140 may be configured to imply a segmentation of anobject instance in a captured image even if that object instance isoccluded by other object instances. The object tracking system 140 mayadditionally be configured to determine which of the pixels associatedwith an object instance correspond with an occluded portion of thatobject instance. This process is generally referred to as “amodalsegmentation” in that the segmentation process takes into considerationthe whole of a physical object even if parts of the physical object arenot necessarily perceived, for example, received images captured by theimage capture devices 114/115. Amodal segmentation may be particularlyadvantageous when performing identity recognition and in a trackingsystem 140 configured for multi-object tracking.

Loss of visual contact is to be expected when tracking an object inmotion through a physical environment. A tracking system 140 basedprimarily on visual inputs (e.g., images captured by image capturedevices 114/115) may lose a track on an object when visual contact islost (e.g., due to occlusion by another object or by the object leavinga FOV of an image capture device 114/115). In such cases, the trackingsystem 140 may become uncertain of the object's location and therebydeclare the object lost. Human pilots generally do not have this issue,particularly in the case of momentary occlusions, due to the notion ofobject permanence. Object permanence assumes that, given certainphysical constraints of matter, an object cannot suddenly disappear orinstantly teleport to another location. Based on this assumption, if itis clear that all escape paths would have been clearly visible, then anobject is likely to remain in an occluded volume. This situation is mostclear when there is single occluding object (e.g., boulder) on flatground with free space all around. If a tracked object in motionsuddenly disappears in the captured image at a location of anotherobject (e.g., the bolder), then it can be assumed that the objectremains at a position occluded by the other object and that the trackedobject will emerge along one of one or more possible escape paths. Insome embodiments, the tracking system 140 may be configured to implementan algorithm that bounds the growth of uncertainty in the trackedobject's location given this concept. In other words, when visualcontact with a tracked object is lost at a particular position, thetracking system 140 can bound the uncertainty in the object's positionto the last observed position and one or more possible escape pathsgiven a last observed trajectory. A possible implementation of thisconcept may include generating, by the tracking system 140, an occupancymap that is carved out by stereo and the segmentations with a particlefilter on possible escape paths.

Unmanned Aerial Vehicle—Example System

A UAV 100, according to the present teachings, may be implemented as anytype of UAV. A UAV, sometimes referred to as a drone, is generallydefined as any aircraft capable of controlled flight without a humanpilot onboard. UAVs may be controlled autonomously by onboard computerprocessors or via remote control by a remotely located human pilot.Similar to an airplane, UAVs may utilize fixed aerodynamic surfacesalong with a propulsion system (e.g., propeller, jet, etc.) to achievelift. Alternatively, similar to helicopters, UAVs may directly use apropulsion system (e.g., propeller, jet, etc.) to counter gravitationalforces and achieve lift. Propulsion-driven lift (as in the case ofhelicopters) offers significant advantages in certain implementations,for example, as a mobile filming platform, because it allows forcontrolled motion along all axes.

Multi-rotor helicopters, in particular quadcopters, have emerged as apopular UAV configuration. A quadcopter (also known as a quadrotorhelicopter or quadrotor) is a multi-rotor helicopter that is lifted andpropelled by four rotors. Unlike most helicopters, quadcopters use twosets of two fixed-pitch propellers. A first set of rotors turnsclockwise, while a second set of rotors turns counter-clockwise. Inturning opposite directions, a first set of rotors may counter theangular torque caused by the rotation of the other set, therebystabilizing flight. Flight control is achieved through variation in theangular velocity of each of the four fixed-pitch rotors. By varying theangular velocity of each of the rotors, a quadcopter may perform preciseadjustments in its position (e.g., adjustments in altitude and levelflight left, right, forward and backward) and orientation, includingpitch (rotation about a first lateral axis), roll (rotation about asecond lateral axis), and yaw (rotation about a vertical axis). Forexample, if all four rotors are spinning (two clockwise, and twocounter-clockwise) at the same angular velocity, the net aerodynamictorque about the vertical yaw axis is zero. Provided the four rotorsspin at sufficient angular velocity to provide a vertical thrust equalto the force of gravity, the quadcopter can maintain a hover. Anadjustment in yaw may be induced by varying the angular velocity of asubset of the four rotors thereby mismatching the cumulative aerodynamictorque of the four rotors. Similarly, an adjustment in pitch and/or rollmay be induced by varying the angular velocity of a subset of the fourrotors but in a balanced fashion such that lift is increased on one sideof the craft and decreased on the other side of the craft. An adjustmentin altitude from hover may be induced by applying a balanced variationin all four rotors, thereby increasing or decreasing the verticalthrust. Positional adjustments left, right, forward, and backward may beinduced through combined pitch/roll maneuvers with balanced appliedvertical thrust. For example, to move forward on a horizontal plane, thequadcopter would vary the angular velocity of a subset of its fourrotors in order to perform a pitch forward maneuver. While pitchingforward, the total vertical thrust may be increased by increasing theangular velocity of all the rotors. Due to the forward pitchedorientation, the acceleration caused by the vertical thrust maneuverwill have a horizontal component and will therefore accelerate the craftforward on a horizontal plane.

FIG. 26 shows a diagram of an example UAV system 2600 including variousfunctional system components that may be part of a UAV 100, according tosome embodiments. UAV system 2600 may include one or more means forpropulsion (e.g., rotors 2602 and motor(s) 2604), one or more electronicspeed controllers 2606, a flight controller 2608, a peripheral interface2610, processor(s) 2612, a memory controller 2614, a memory 2616 (whichmay include one or more computer readable storage media), a power module2618, a GPS module 2620, a communications interface 2622, audiocircuitry 2624, an accelerometer 2626 (including subcomponents such asgyroscopes), an IMU 2628, a proximity sensor 2630, an optical sensorcontroller 2632 and associated optical sensor(s) 2634, a mobile deviceinterface controller 2636 with associated interface device(s) 2638, andany other input controllers 2640 and input device(s) 2642, for example,display controllers with associated display device(s). These componentsmay communicate over one or more communication buses or signal lines asrepresented by the arrows in FIG. 26 .

UAV system 2600 is only one example of a system that may be part of aUAV 100. A UAV 100 may include more or fewer components than shown insystem 2600, may combine two or more components as functional units, ormay have a different configuration or arrangement of the components.Some of the various components of system 2600 shown in FIG. 26 may beimplemented in hardware, software or a combination of both hardware andsoftware, including one or more signal processing and/or applicationspecific integrated circuits. Also, UAV 100 may include an off-the-shelfUAV (e.g., a currently available remote-controlled quadcopter) coupledwith a modular add-on device (for example, one including componentswithin outline 2690) to perform the innovative functions described inthis disclosure.

As described earlier, the means for propulsion 2602-2604 may comprisefixed-pitch rotors. The means for propulsion may also includevariable-pitch rotors (for example, using a gimbal mechanism), avariable-pitch jet engine, or any other mode of propulsion having theeffect of providing force. The means for propulsion 2602-2604 mayinclude a means for varying the applied thrust, for example, via anelectronic speed controller 2606 varying the speed of each fixed-pitchrotor.

Flight controller 2608 may include a combination of hardware and/orsoftware configured to receive input data (e.g., sensor data from imagecapture devices 2634, and or generated trajectories form an autonomousnavigation system 120), interpret the data and output control commandsto the propulsion systems 2602-2606 and/or aerodynamic surfaces (e.g.,fixed wing control surfaces) of the UAV 100. Alternatively, or inaddition, a flight controller 2608 may be configured to receive controlcommands generated by another component or device (e.g., processors 2612and/or a separate computing device), interpret those control commandsand generate control signals to the propulsion systems 2602-2606 and/oraerodynamic surfaces (e.g., fixed wing control surfaces) of the UAV 100.In some embodiments, the previously mentioned navigation system 120 ofthe UAV 100 may comprise the flight controller 2608 and/or any one ormore of the other components of system 2600. Alternatively, the flightcontroller 2608 shown in FIG. 26 may exist as a component separate fromthe navigation system 120, for example, similar to the flight controller160 shown in FIG. 2 .

Memory 2616 may include high-speed random access memory and may alsoinclude non-volatile memory, such as one or more magnetic disk storagedevices, flash memory devices, or other non-volatile solid-state memorydevices. Access to memory 2616 by other components of system 2600, suchas the processors 2612 and the peripherals interface 2610, may becontrolled by the memory controller 2614.

The peripherals interface 2610 may couple the input and outputperipherals of system 2600 to the processor(s) 2612 and memory 2616. Theone or more processors 2612 run or execute various software programsand/or sets of instructions stored in memory 2616 to perform variousfunctions for the UAV 100 and to process data. In some embodiments,processors 2612 may include general central processing units (CPUs),specialized processing units such as graphical processing units (GPUs)particularly suited to parallel processing applications, or anycombination thereof. In some embodiments, the peripherals interface2610, the processor(s) 2612, and the memory controller 2614 may beimplemented on a single integrated chip. In some other embodiments, theymay be implemented on separate chips.

The network communications interface 2622 may facilitate transmissionand reception of communications signals often in the form ofelectromagnetic signals. The transmission and reception ofelectromagnetic communications signals may be carried out over physicalmedia such as copper wire cabling or fiber optic cabling, or may becarried out wirelessly, for example, via a radiofrequency (RF)transceiver. In some embodiments, the network communications interfacemay include RF circuitry. In such embodiments, RF circuitry may convertelectrical signals to/from electromagnetic signals and communicate withcommunications networks and other communications devices via theelectromagnetic signals. The RF circuitry may include well-knowncircuitry for performing these functions, including, but not limited to,an antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. The RFcircuitry may facilitate transmission and receipt of data overcommunications networks (including public, private, local, and widearea). For example, communication may be over a wide area network (WAN),a local area network (LAN), or a network of networks such as theInternet. Communication may be facilitated over wired transmission media(e.g., via Ethernet) or wirelessly. Wireless communication may be over awireless cellular telephone network, a wireless local area network (LAN)and/or a metropolitan area network (MAN), and other modes of wirelesscommunication. The wireless communication may use any of a plurality ofcommunications standards, protocols and technologies, including, but notlimited to, Global System for Mobile Communications (GSM), Enhanced DataGSM Environment (EDGE), high-speed downlink packet access (HSDPA),wideband code division multiple access (W-CDMA), code division multipleaccess (CDMA), time division multiple access (TDMA), Bluetooth, WirelessFidelity (Wi-Fi) (e.g., IEEE 802.11n and/or IEEE 802.11ac), voice overInternet Protocol (VoIP), Wi-MAX, or any other suitable communicationprotocols.

The audio circuitry 2624, including the speaker and microphone 2650, mayprovide an audio interface between the surrounding environment and theUAV 100. The audio circuitry 2624 may receive audio data from theperipherals interface 2610, convert the audio data to an electricalsignal, and transmit the electrical signal to the speaker 2650. Thespeaker 2650 may convert the electrical signal to human-audible soundwaves. The audio circuitry 2624 may also receive electrical signalsconverted by the microphone 2650 from sound waves. The audio circuitry2624 may convert the electrical signal to audio data and transmit theaudio data to the peripherals interface 2610 for processing. Audio datamay be retrieved from and/or transmitted to memory 2616 and/or thenetwork communications interface 2622 by the peripherals interface 2610.

The I/O subsystem 2660 may couple input/output peripherals of UAV 100,such as an optical sensor system 2634, the mobile device interface 2638,and other input/control devices 2642, to the peripherals interface 2610.The I/O subsystem 2660 may include an optical sensor controller 2632, amobile device interface controller 2636, and other input controller(s)2640 for other input or control devices. The one or more inputcontrollers 2640 receive/send electrical signals from/to other input orcontrol devices 2642.

The other input/control devices 2642 may include physical buttons (e.g.,push buttons, rocker buttons, etc.), dials, touch screen displays,slider switches, joysticks, click wheels, and so forth. A touch screendisplay may be used to implement virtual or soft buttons and one or moresoft keyboards. A touch-sensitive touch screen display may provide aninput interface and an output interface between the UAV 100 and a user.A display controller may receive and/or send electrical signals from/tothe touch screen. The touch screen may display visual output to a user.The visual output may include graphics, text, icons, video, and anycombination thereof (collectively termed “graphics”). In someembodiments, some or all of the visual output may correspond touser-interface objects, further details of which are described below.

A touch sensitive display system may have a touch-sensitive surface,sensor or set of sensors that accepts input from the user based onhaptic and/or tactile contact. The touch sensitive display system andthe display controller (along with any associated modules and/or sets ofinstructions in memory 2616) may detect contact (and any movement orbreaking of the contact) on the touch screen and convert the detectedcontact into interaction with user-interface objects (e.g., one or moresoft keys or images) that are displayed on the touch screen. In anexemplary embodiment, a point of contact between a touch screen and theuser corresponds to a finger of the user.

The touch screen may use liquid crystal display (LCD) technology, orlight emitting polymer display (LPD) technology, although other displaytechnologies may be used in other embodiments. The touch screen and thedisplay controller may detect contact and any movement or breakingthereof using any of a plurality of touch sensing technologies now knownor later developed, including, but not limited to, capacitive,resistive, infrared, and surface acoustic wave technologies, as well asother proximity sensor arrays or other elements for determining one ormore points of contact with a touch screen.

The mobile device interface device 2638 along with mobile deviceinterface controller 2636 may facilitate the transmission of databetween a UAV 100 and other computing devices such as a mobile device104. According to some embodiments, communications interface 2622 mayfacilitate the transmission of data between UAV 100 and a mobile device104 (for example, where data is transferred over a Wi-Fi network).

UAV system 2600 also includes a power system 2618 for powering thevarious components. The power system 2618 may include a power managementsystem, one or more power sources (e.g., battery, alternating current(AC), etc.), a recharging system, a power failure detection circuit, apower converter or inverter, a power status indicator (e.g., alight-emitting diode (LED)) and any other components associated with thegeneration, management and distribution of power in computerized device.

UAV system 2600 may also include one or more image capture devices 2634.Image capture devices 2634 may be the same as the image capture device114/115 of UAV 100 described with respect to FIG. 1A. FIG. 26 shows animage capture device 2634 coupled to an image capture controller 2632 inI/O subsystem 2660. The image capture device 2634 may include one ormore optical sensors. For example, image capture device 2634 may includea charge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. The optical sensors of image capture devices2634 receive light from the environment, projected through one or morelens (the combination of an optical sensor and lens can be referred toas a “camera”) and converts the light to data representing an image. Inconjunction with an imaging module located in memory 2616, the imagecapture device 2634 may capture images (including still images and/orvideo). In some embodiments, an image capture device 2634 may include asingle fixed camera. In other embodiments, an image capture device 2640may include a single adjustable camera (adjustable using a gimbalmechanism with one or more axes of motion). In some embodiments, animage capture device 2634 may include a camera with a wide-angle lensproviding a wider FOV. In some embodiments, an image capture device 2634may include an array of multiple cameras providing up to a full 360degree view in all directions. In some embodiments, an image capturedevice 2634 may include two or more cameras (of any type as describedherein) placed next to each other in order to provide stereoscopicvision. In some embodiments, an image capture device 2634 may includemultiple cameras of any combination as described above. In someembodiments, the cameras of an image capture device 2634 may be arrangedsuch that at least two cameras are provided with overlapping FOV atmultiple angles around the UAV 100, thereby allowing for stereoscopic(i.e., 3D) image/video capture and depth recovery (e.g., throughcomputer vision algorithms) at multiple angles around UAV 100. Forexample, UAV 100 may include four sets of two cameras each positioned soas to provide a stereoscopic view at multiple angles around the UAV 100.In some embodiments, a UAV 100 may include some cameras dedicated forimage capture of a subject and other cameras dedicated for image capturefor visual navigation (e.g., through visual inertial odometry).

UAV system 2600 may also include one or more proximity sensors 2630.FIG. 26 shows a proximity sensor 2630 coupled to the peripheralsinterface 2610. Alternately, the proximity sensor 2630 may be coupled toan input controller 2640 in the I/O subsystem 2660. Proximity sensors2630 may generally include remote sensing technology for proximitydetection, range measurement, target identification, etc. For example,proximity sensors 2630 may include radar, sonar, and LIDAR.

UAV system 2600 may also include one or more accelerometers 2626. FIG.26 shows an accelerometer 2626 coupled to the peripherals interface2610. Alternately, the accelerometer 2626 may be coupled to an inputcontroller 2640 in the I/O subsystem 2660.

UAV system 2600 may include one or more IMU 2628. An IMU 2628 maymeasure and report the UAV's velocity, acceleration, orientation, andgravitational forces using a combination of gyroscopes andaccelerometers (e.g., accelerometer 2626).

UAV system 2600 may include a global positioning system (GPS) receiver2620. FIG. 26 shows an GPS receiver 2620 coupled to the peripheralsinterface 2610. Alternately, the GPS receiver 2620 may be coupled to aninput controller 2640 in the I/O subsystem 2660. The GPS receiver 2620may receive signals from GPS satellites in orbit around the earth,calculate a distance to each of the GPS satellites (through the use ofGPS software), and thereby pinpoint a current global position of UAV100.

In some embodiments, the software components stored in memory 2616 mayinclude an operating system, a communication module (or set ofinstructions), a flight control module (or set of instructions), alocalization module (or set of instructions), a computer vision module,a graphics module (or set of instructions), and other applications (orsets of instructions). For clarity, one or more modules and/orapplications may not be shown in FIG. 26 .

An operating system (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VxWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

A communications module may facilitate communication with other devicesover one or more external ports 2644 and may also include varioussoftware components for handling data transmission via the networkcommunications interface 2622. The external port 2644 (e.g., UniversalSerial Bus (USB), FIREWIRE, etc.) may be adapted for coupling directlyto other devices or indirectly over a network (e.g., the Internet,wireless LAN, etc.).

A graphics module may include various software components forprocessing, rendering and displaying graphics data. As used herein, theterm “graphics” may include any object that can be displayed to a user,including, without limitation, text, still images, videos, animations,icons (such as user-interface objects including soft keys), and thelike. The graphics module in conjunction with a graphics processing unit(GPU) 2612 may process in real time or near real time, graphics datacaptured by optical sensor(s) 2634 and/or proximity sensors 2630.

A computer vision module, which may be a component of a graphics module,provides analysis and recognition of graphics data. For example, whileUAV 100 is in flight, the computer vision module along with a graphicsmodule (if separate), GPU 2612, and image capture devices(s) 2634 and/orproximity sensors 2630 may recognize and track the captured image of anobject located on the ground. The computer vision module may furthercommunicate with a localization/navigation module and flight controlmodule to update a position and/or orientation of the UAV 100 and toprovide course corrections to fly along a planned trajectory through aphysical environment.

A localization/navigation module may determine the location and/ororientation of UAV 100 and provide this information for use in variousmodules and applications (e.g., to a flight control module in order togenerate commands for use by the flight controller 2608).

Image capture devices(s) 2634, in conjunction with an image capturedevice controller 2632 and a graphics module, may be used to captureimages (including still images and video) and store them into memory2616.

Each of the above identified modules and applications correspond to aset of instructions for performing one or more functions describedabove. These modules (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, and,thus, various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 2616 maystore a subset of the modules and data structures identified above.Furthermore, memory 2616 may store additional modules and datastructures not described above.

Example Computer Processing System

FIG. 27 is a block diagram illustrating an example of a processingsystem 2700 in which at least some operations described in thisdisclosure can be implemented. The example processing system 2700 may bepart of any of the aforementioned devices including, but not limited toUAV 100 and mobile device 104. The processing system 2700 may includeone or more central processing units (“processors”) 2702, main memory2706, non-volatile memory 2710, network adapter 2712 (e.g., networkinterfaces), display 2718, input/output devices 2720, control device2722 (e.g., keyboard and pointing devices), drive unit 2724 including astorage medium 2726, and signal generation device 2730 that arecommunicatively connected to a bus 2716. The bus 2716 is illustrated asan abstraction that represents any one or more separate physical buses,point to point connections, or both connected by appropriate bridges,adapters, or controllers. The bus 2716, therefore, can include, forexample, a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus (also called “Firewire”).A bus may also be responsible for relaying data packets (e.g., via fullor half duplex wires) between components of the network appliance, suchas the switching fabric, network port(s), tool port(s), etc.

In various embodiments, the processing system 2700 may be a servercomputer, a client computer, a personal computer (PC), a user device, atablet PC, a laptop computer, a personal digital assistant (PDA), acellular telephone, an iPhone, an iPad, a Blackberry, a processor, atelephone, a web appliance, a network router, switch or bridge, aconsole, a hand-held console, a (hand-held) gaming device, a musicplayer, any portable, mobile, hand-held device, or any machine capableof executing a set of instructions (sequential or otherwise) thatspecify actions to be taken by the computing system.

While the main memory 2706, non-volatile memory 2710, and storage medium2726 (also called a “machine-readable medium”) are shown to be a singlemedium, the term “machine-readable medium” and “storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store one or more sets of instructions 2728. The term“machine-readable medium” and “storage medium” shall also be taken toinclude any medium that is capable of storing, encoding, or carrying aset of instructions for execution by the computing system and that causethe computing system to perform any one or more of the methodologies ofthe presently disclosed embodiments.

In general, the routines executed to implement the embodiments of thedisclosure, may be implemented as part of an operating system or aspecific application, component, program, object, module, or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions (e.g., instructions 2704,2708, 2728) set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessing units or processors 2702, cause the processing system 2700 toperform operations to execute elements involving the various aspects ofthe disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable (storage) media include recordable typemedia such as volatile and non-volatile memory devices 2710, floppy andother removable disks, hard disk drives, optical disks (e.g., CompactDisk Read-Only Memory (CD ROMS), Digital Versatile Disks (DVDs)), andtransmission type media such as digital and analog communication links.

The network adapter 2712 enables the processing system 2700 to mediatedata in a network 2714 with an entity that is external to the processingsystem 2700, such as a network appliance, through any known and/orconvenient communications protocol supported by the processing system2700 and the external entity. The network adapter 2712 can include oneor more of a network adaptor card, a wireless network interface card, arouter, an access point, a wireless router, a switch, a multilayerswitch, a protocol converter, a gateway, a bridge, bridge router, a hub,a digital media receiver, and/or a repeater.

The network adapter 2712 can include a firewall which can, in someembodiments, govern and/or manage permission to access/proxy data in acomputer network, and track varying levels of trust between differentmachines and/or applications. The firewall can be any number of moduleshaving any combination of hardware and/or software components able toenforce a predetermined set of access rights between a particular set ofmachines and applications, machines and machines, and/or applicationsand applications, for example, to regulate the flow of traffic andresource sharing between these varying entities. The firewall mayadditionally manage and/or have access to an access control list whichdetails permissions including, for example, the access and operationrights of an object by an individual, a machine, and/or an application,and the circumstances under which the permission rights stand.

As indicated above, the techniques introduced here may be implementedby, for example, programmable circuitry (e.g., one or moremicroprocessors), programmed with software and/or firmware, entirely inspecial-purpose hardwired (i.e., non-programmable) circuitry, or in acombination or such forms. Special-purpose circuitry can be in the formof, for example, one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field-programmable gatearrays (FPGAs), etc.

Note that any of the embodiments described above can be combined withanother embodiment, except to the extent that it may be stated otherwiseabove or to the extent that any such embodiments might be mutuallyexclusive in function and/or structure.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Accordingly, the specification and drawings are to be regardedin an illustrative sense rather than a restrictive sense.

What is claimed is:
 1. An unmanned aerial vehicle (UAV) configured forautonomous flight through a physical environment, the UAV including: anapplication programming interface (API) configured to: expose parametersof a behavioral objective to a third-party application, wherein thebehavioral objective defines a goal or target affecting autonomouscontrol of the UAV; process a call received from the third-partyapplication, the call configured to set or modify parameters of thebehavioral objective; transmit an instruction to set or modify theparameters of the behavioral objective in a multi-objective trajectorygeneration process; a motion planning system configured to: set ormodify the parameters of the behavioral objective in the multi-objectivetrajectory generation process responsive to receiving the instruction;receive perception data based on images of the physical environment;feed the perception data into the multi-objective trajectory generationprocess to generate a trajectory that satisfies the behavioral objectivein view of another behavioral objective or constraint; and a flightcontroller configured to cause the UAV to autonomously maneuver based onthe trajectory.
 2. The UAV of claim 1, further comprising: an imagecapture device configured to capture the images of the physicalenvironment.
 3. The UAV of claim 1, wherein the motion planning systemis configured to: continually receive additional perception data basedon additional images of the physical environment; and responsivelyperform the trajectory generation process with the additional perceptiondata.
 4. The UAV of claim 1, wherein the one or more parameters of thebehavioral objective include one or more of: a target to be achieved bythe trajectory generation process, a dead-zone region about the targetupon which the trajectory generation process will not operate to adjustthe proposed trajectory, or a weighting factor defining a level ofimpact of the behavioral objective on the proposed trajectory in view ofthe another behavioral objective or constraint.
 5. The UAV of claim 1,wherein the another behavioral objective is configured to constrain theproposed trajectory to avoid a collision between the UAV and otherobjects in the physical environment, and/or conform with dynamiclimitations of the UAV.
 6. The UAV of claim 1, further comprising: amemory unit with the third-party application stored thereon, thethird-party application configured to generate the call to the API toset the one or more parameters.
 7. The UAV of claim 1, furthercomprising: a wireless communication interface through which tocommunicate with a wireless communication device; wherein, the call tothe API is generated by the third-party application based on indicationof a user input at a mobile device, received via the wirelesscommunication device.
 8. An apparatus, comprising: one or more memoryunits storing instructions that, when executed by one or moreprocessors, cause the one or more processors to: expose, via anavigation application programming interface (API) of an unmanned aerialvehicle (UAV), parameters of a behavioral objective to a third-partyapplication, wherein the behavioral objective defines a goal or targetaffecting autonomous control of the UAV; receive, at the API from thethird-party application, a call setting or modifying parameters of thebehavioral objective; and set or modify the parameters of the behavioralobjective in a multi-objective trajectory generation process, whereinthe multi-objective trajectory generation process is configured togenerate a proposed trajectory that satisfies the behavioral objectivein view of another behavioral objective or constraint.
 9. The apparatusof claim 8, wherein the instructions, when executed by the one or moreprocessors, further cause the one or more processors to: cause the UAVto autonomously maneuver through the physical environment based on theproposed trajectory.
 10. The apparatus of claim 8, wherein thebehavioral objective is any of a navigation objective or an imagecapture objective.
 11. The apparatus of claim 8, wherein the behavioralobjective is defined relative to one or more of the physicalenvironment, the UAV, a physical object located in the physicalenvironment, or a captured image of the physical environment.
 12. Theapparatus of claim 8, wherein the behavioral objective is definedrelative to a semantic understanding of the physical environment tomaintain saliency in images captured by an image capture device coupledto the UAV.
 13. The apparatus of claim 8, wherein causing the UAV toautonomously maneuver through the physical environment based on theproposed trajectory includes: generating, by the computer system,control commands configured to cause the UAV to autonomously maneuveralong the proposed trajectory.
 14. The apparatus of claim 8, wherein theone or more parameters of the behavioral objective include any of: atarget to be achieved by the trajectory generation process, a dead-zoneregion about the target upon which the trajectory generation processwill not operate to adjust the proposed trajectory, or a weightingfactor defining a level of impact of the behavioral objective on theproposed trajectory in view of the another behavioral objective orconstraint.
 15. The apparatus of claim 8, wherein the another behavioralobjective is defined based on another API call.
 16. The apparatus ofclaim 8, wherein the another behavioral objective is configured toconstrain the proposed trajectory to: avoid a collision between the UAVand other objects in the physical environment; or conform with dynamiclimitations of the UAV.
 17. The apparatus of claim 8, wherein the one ormore parameters include a weighting factor defining a level of impact ofthe behavioral objective on the proposed trajectory in view of theanother behavioral objective or constraint.
 18. A method comprising:exposing, via a navigation application programming interface (API) of anunmanned aerial vehicle (UAV), parameters of a behavioral objective to athird-party application, wherein the behavioral objective defines a goalor target affecting autonomous control of the UAV; receiving, at the APIfrom the third-party application, a call setting or modifying parametersof the behavioral objective; and setting or modifying the parameters ofthe behavioral objective in a multi-objective trajectory generationprocess, wherein the multi-objective trajectory generation process isconfigured to generate a proposed trajectory that satisfies thebehavioral objective in view of another behavioral objective orconstraint.
 19. The method of claim 18, further comprising: causing theUAV to autonomously maneuver through the physical environment based onthe proposed trajectory.
 20. The method of claim 18, wherein thebehavioral objective is any of a navigation objective or an imagecapture objective.